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8th  Annual  Systems  Engineering  Conference 
“Focusing  on  Mission  Areasy  Net-Centric  Operations 
and  Supportability  of  Defense  Systems 99 

San  Diego,  CA 

24-27  October  2005 


Agenda 


Tuesday.  25  October  2005 


Open  Remarks:  by  Mr.  Bob  Rassa,  Director,  Systems  Supportability,  Raytheon;  Chair,  Systems  Engineering  Division,  NDIA 
Keynote  Address:  by  Mr.  John  Landon,  Deputy  Assistant  Secretary  of  Defense  (Nil),  C3ISR  &  IT  Acquisition 

Plenary  Session  -  Revitalization  of  Systems  Engineering  Within  DoD: 

•  State  of  Systems  Engineering  within  DoDs,  Mr.  Mark  D.  Schaeffer,  Deputy  Director,  Systems  Engineering,  OUSD  (AT&L) 

•  USAF  Systems  Engineering  Initiatives,  Mr.  Terry  Jaggers,  SAF/AQR  (Science  &  Technology  &  Engineering) 

•  System  Engineering  Re-vitalization  within  DoN  Status,  Mr.  Carl  Siel,  ASN(RDA)  Chief  Engineer 

•  Army  SE  Overview,  Mr.  Douglas  K.  Wiltsie,  Assistant  Deputy,  Acquisition  and  Systems  Management,  Office  of  the  Assistant  Secretary  of  the  Army, 
Acquisition  Logistics  and  Technology 

•  “Implementation  of  ESE/A”,  Mr.  Kelly  A.  Miller,  NSA/CSS  CSE 

Luncheon  Keynote  Speaker:  by  Mr.  Gregory  Shelton,  Corporate  Vice  President,  Engineering,  Technology,  Manufacturing  and  Quality,  Raytheon  Company 
Tracks  1  &  2  -  Systems  Engineering  Effectiveness: 

•  Technical  Planning  for  Acquisition  Programs:  An  OSD  Perspective,  Col  Warren  Anderson,  OUSD  (AT&L)  Defense  Systems 

•  Implementation  of  Policy  Requiring  Systems  Engineering  Plans  for  Air  Force  Programs  -  Results  and  Implications,  Mr.  Kevin  Kemper,  Air  Force  Materiel 
Command 

•  Systems  Engineering  Revitalization  at  SPAWAR  Systems  Center  Charleston,  Mr.  Michael  T.  Kutch,  Jr.,  SPAWAR  Systems  Center 

•  Systems  Engineering  for  Software  Assurance,  Ms.  Kristen  Baldwin,  OUSD  (AT&L)  Defense  Systems 

•  Revitalization  of  Systems  Engineering:  Past,  Present  and  Future,  Ms.  Karen  B.  Bausman,  Air  Force  Center  for  Systems  Engineering 

•  Enabling  Technology  Readiness  Assessments  (TRAs)  with  Systems  Engineering,  Dr.  Jay  Mandelbaum,  Institute  for  Defense  Analyses 

•  A  Taxonomy  of  Operational  Risks,  Mr.  Brian  Gallagher,  Software  Engineering  Institute 

•  A  Method  for  Reasoning  About  an  Acquisition  Strategy,  Mr.  Joseph  Elm,  Software  Engineerin  Institute 

•  WBS-Based  Approach  to  Understanding  and  Predicting  Program  Risk,  Bruce  M.  Heim,  DCMA,  Boeing  Long  Beach 

•  Program  Support:  Perspectives  on  Technical  Planning  and  Execution,  Mr.  Dave  Castellano,  OUSD  (AT&L)  Systems  Engineering 

Track  3  -  Test  &  Evaluation  in  Systems  Engineering: 

•  Interweaving  Test  and  Evaluation  Throughout  the  Systems  Engineering  Process  -  Presentation  and  Paper,  Mr.  Josh  Tribble,  AVW  Technologies 
Track  4  -  Net  Centric  Operations: 

•  Net-Centricity  &  Net-Ready  -  Beyond  Technical  Interoperability  &  C4ISR,  Mr.  Jack  Zavin,  ASD(NII),  DoD  CIO/A&I  Directorate 

•  A  Strategy  for  Managing  Development  and  Certification  of  Net-Centric  Services  within  the  Global  Information  Grid,  Mr.  Bernal  Allen,  DISA,  GE  4 

•  Next  Generation  Enterprise  Information  Management  Appliances,  Mr.  Michael  Lindow,  The  MITRE  Corp. 

Track  5  -  Logistics: 

•  Logistics  Transforming:  Achieving  Knowledge-Enabled  Logistics,  Mr.  Jerry  Beck,  OSD  Office  of  ADUSD(LPP) 

•  Condition  Based  Logistics,  Mr.  Ron  Wagner,  CoBaLt  Technology 

•  System  Supportability  and  Life  Cycle  Product  Support:  A  Systems  Perspective,  Dinesh  Verma,  Stevens  Institute  of  Technolog 

•  The  Management  of  Logistics  in  Large  Scale  Inventory  Systems  to  Support  Weapon  System  Maintenance,  Mr.  Eugene  A.  Beardslee,  SAIC 

Track  7  -  Systems  Safety: 
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•  System  Safety  in  Systems  Engineering  DAU  Continuous  Learning  Module,  Ms.  Amanda  Zarecky,  Booz  Allen  Hamilton 

•  Enabling  System  Safety  Through  Technical  Excellence,  Col  Warren  Anderson,  OUSD  (AT&L)  Defense  Systems 

•  Applying  CMMI  to  System  Safety,  Mr.  Tom  Pfitzer,  APT  Research,  Inc. 

•  System  Safety  Engineering:  An  Overview  for  Engineers  and  Managers,  Mr.  Pat  L.  Clemens,  APT  Research,  Inc. 

•  Using  MIL-STD-882D  to  Integrate  ESOH  into  SE,  Mr.  Sherman  G.  Forbes,  USAF  -  SAF/AQRE 

Track  8  -  Software  Supportabilitv: 

•  The  Proper  Specification  of  Requirements,  Mr.  A1  Florence,  The  MITRE  Corporation 

•  C-17  Software  Development  Process,  John  R.  Allen,  The  Boeing  Company  4 

•  Successful  Verification  and  Validation  Based  on  the  CMMI  Model,  Mr.  Tim  Olson,  Quality  Improvement  Consultants,  Inc. 

•  “Automated  Software  Testing  Increases  Test  Quality  and  Coverage  Resulting  in  Improved  Software  Reliability.”,  Mr.  Frank  Salvatore,  High  Performance 
Technologies,  Inc. 

•  Software  Supportability:  A  Software  Engineering  Perspective,  Ms.  Stephany  Bellomo,  SAIC 


Wednesday.  26  October  2005 


Tracks  1,  2  &  3  -  Systems  Engineering  Effectiveness: 

•  Decision  Analysis  and  Resolution,  Mr  Robert  Trifiletti,  Jr.,  US  Army  ARDEC 

•  Defining  System  Development  Lifecycles  to  Plan  and  Manage  Projects  Effectively,  Mr.  Bruce  A.  Boyd,  The  Boeing  Company 

•  Systems  Engineering,  Program  Management  conjoined  Disciplines  over  the  Project  Life  Cycle,  Mr.  William  Lyders,  ASSETT,  Inc. 

•  Tailoring  USAF  Systems  Engineering  for  the  Life  Cycle:  One  Shape,  Multiple  Dimensions,  Mr.  Jeff  Loren,  MTC  Technologies,  Inc.  (SAF/AQRE) 

•  Architecture-Based  Systems  Engineering  and  Integration,  Dr.  Rick  Habayeb,  Virginia  Polytechnic  Institute  &  State  University 

•  A  Complementary  Approach  to  Enterprise  Systems  Engineering,  Dr.  Brian  White,  The  MITRE  Corporation 

•  Implementing  Systems  Engineering  Processes  to  Balance  Cost  and  Technical  Performance,  Dr.  Mary  Anne  Herndon,  Transdyne  Corporation 

•  Program  Support:  Perspectives  on  Technical  Planning  and  Execution,  Mr.  Dave  Castellano,  OUSD  (AT&L)  Systems  Engineering 

•  Application  of  Risk  Management  in  a  Net-Centric  Environment,  Ms.  Rebecca  M.  Cowen-Hirsch,  DISA 

•  “Requirements  Management  Tips  and  Tricks”,  Mr.  Frank  Salvatore,  High  Performance  Technologies,  Inc. 

•  Engineering  and  Implementing  Raytheon  Missile  Systems  Engineering  Design  to  Cost  Metric  -  Presentation  and  Paper,  Mr.  Edward  Casey,  Raytheon  Missile 
Systems 

•  System  Engineering  Metrics,  Mr.  James  Miller,  Air  Foce  Materiel  Command 

•  Technical  Performance  Measures,  Mr.  Jim  Oakes,  BAE  Systems 

•  TurboTax®  for  Systems  EngineerinTurboTax®  for  Systems  Engineering,  Michael  T.  Kutch,  Jr.,  SPAWAR 

•  A  Practical  Application  of  A  Practical  Application  of  the  Non- Advocate  Review,  Mr.  Bruce  Nishime,  The  Boeing  Company 

•  Systems  Engineering  and  the  Software  Laws  of  Thermodynamics,  Dr.  Thomas  F.  Christian  Jr.,  402  SMXG 

•  Unmanned  Aerial  Vehicle  Survivability  Influence  on  System  Life  Cycle  Cost,  Mr.  Chuck  Pedriani,  SURVICE  Engineering 

•  Effective  SE  Metrics  Tailored  to  the  Acquisition  Life  Cycle,  Ms.  Laura  Trioilia,  US  Army  ARDEC 

•  Innovative  Procurement  Strategies,  Mr.  David  Eiband,  Defense  Acquisition  University 

•  Next  Generation  Combat  Systems  -  An  Overview  of  Key  Development  Concepts,  Mr.  Matthew  Montoya,  The  JHU  Applied  Physics  Laboratory  Mr.  Edward 
Casey,  Raytheon  Missile  Systems 

•  Converting  High-Level  Systems  Engineering  Policy  to  a  Workable  Program,  Mr.  James  Miller,  Air  Force  Materiel  Command 

•  AFRL  Systems  Engineering  Initiative  -  Risk  Managment  for  Science  and  Technology,  Mr.  William  Nolte,  USAF-AFRL 

•  System  Engineered  Research  and  Development  Magement,  Dr.  Steven  Ligon,  SAIC 

•  The  Return  of  Discipline,  Ms.  Jacqueline  Townsend,  Air  Force  Materiel  Command 

Track  4  -  Net  Centric  Operations: 

•  Testing  Net-Centric  Systems  of  Systems:  Applying  Lessons  Learned  from  Distributed  Simulation,  Mr.  Doug  Flournoy,  The  MITRE  Corp. 

•  A  Multi-Mission  Network  Centric  Warfare  Platform,  Peder  Jungck,  CloudSheild  Technologies 

•  Challenges  Challenges  in  Development  of  System  of  Systems  (SoS)  Architectures  in  a  Net  Centric  Environment,  Dr.  Abraham  Meilich,  Lockheed  Martin 

•  Matrix  Mapping  Tool  (MMT),  Dr.  Judith  Dahmann,  AT&L/DS  MITRE 

Track  5  -  Logistics: 

•  Defense  Logistics  as  Chaos  Theory,  Mr.  John  Sells,  Tobyhanna  Army  Depot 

•  Process  for  Evaluating  LogisticProcess  for  Evaluating  Logistics  Readiness  Levels  (LRLs)  for  Acquisition  Systems,  Ms.  Elizabeth  Broadus,  Booz  Allen 
Hamilton,  Inc. 

•  The  Management  of  Logistics  in  Large  Scale  Inventory  Systems  to  Support  Weapon  System  Maintenance,  Mr.  Eugene  A.  Beardslee,  SAIC 

•  System  of  Systems  Analysis  of  Future  Combat  Systems  Sustainment  Requirements,  Mr.  Ivan  W.  Wolnek,  The  Boeing  Company 

•  Readiness  &  Supportability  Program  Readiness  &  Supportability  Programs,  Mr.  Robert  M.  Cranwell,  Sandia  National  Laboratories  (SNL) 

•  Data  Management  in  a  Performance  Based  Logistics  Environment,  Denise  Duncan,  LMI 

Track  5  -  Best  Practices  &  Standardization: 

•  CMMI  for  Services,  Mr.  Juan  Ceva,  Raytheon  Company 

•  Out  of  the  Ordinary:  Finding  Hidden  Threats  by  Analyzing  Unusual  Behavior,  Mr.  John  Hollywood,  RAND 
Track  6  -  Modeling  &  Simulation: 

•  Improving  M&S  Support  to  Acquisition:  A  Progress  Report  on  Development  of  the  Acquisition  M&S  Master  Plan,  Mr.  Jim  Hollenbach,  Simulation  Strategies, 
Inc. 
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•  Next  Generation  Manufacturing  Technology  Initiative  and  the  Model  -  Based  Enterprise,  Mr.  Richard  Neal  -  IMTI 

•  Problem  Space  Modeling:  A  Dynamic  Future  for  Requirements  Analysis,  Mr.  Jeffrey  O.  Grady,  JOG  System  Engineering,  Inc. 

•  Systems  Modeling  Language  Systems  Modeling  Language  (SysML)  Overview  &  Update,  Rick  Steiner,  Raytheon  Company 

•  Data  Management  Support  for  Modeling  and  Simulation,  Mr.  Denise  Duncan,  LMI 

•  Digital  Data  Management  an  Update,  Ms.  Cynthia  C.  Hauer,  Millennium  Data  Management,  Inc. 

•  The  Use  of  Simulation  in  the  Management  of  Logistics  in  Large  Scale  Inventory  Systems  to  Support  Weapon  System  Maintenance,  Mr.  Eugene  A.  Beardslee, 
SAIC 

Track  7  -  System  Safety: 

•  Mission  Sustainment  Through  Environment,  Safety,  and  Occupational  Health  (ESOH)  Risk  Management,  Ms.  Trish  Huheey,  ODUSD  (I&E) 

•  Lessons  Learned  with  the  Application  of  MIL-STD-882D  at  the  Weapon  System  Explosives  Safety  Review  Board,  Ms.  Mary  Ellen  Caro,  Ordnance  Safety  & 
Security  Activity 

•  Industry  Perspectives  and  Identified  Barriers  to  the  Use  of  MIL-STD-882D  for  Integrating  ESOH  Considerations  into  Systems,  Mr.  Jon  Derickson,  BAE 
Systems 

•  System  Safety  in  Systems  Engineering  Process,  Dr.  Ray  C.  Terry,  SURVICE  Engineering  Company 

•  Enabling  Army  Level  Risk  Mitigation,  Mr.  Bill  Edmonds,  US  Army  Combat  Readiness  Center 

•  Evolution  of  MIL-STD-882E,  Mr.  Robert  McAllister,  US  Air  Force  Materiel  Command 

•  Integrating  MIL- STD-8 82  System  Safety  Products  into  the  Concurrent  Engineering  Approach  to  System  Design,  Build,  Test,  and  Delivery  of  Submarine 
Systems  At  Electric  Boat,  Mr.  Ricky  Milnarik,  General  Dynamics 

Track  8  -  Legacy  Systems  Sustainment: 

•  Sustaining  Software-Intensive  Systems  -  A  Conundrum,  Ms.  Mary  Ann  Lapham,  Carnegie  Mellon  Software  Engineering  Institute 

•  Algorithm  Description  Documentation  and  Validation  Process,  Mr.  Mike  Bailey,  Raytheon  Company 

•  ATSRAC:  Background,  Results  and  Future  Impact  on  the  Aviation  Industry,  Mr.  Kent  V.  Hollinger,  The  MITRE  Corp. 

•  Jammer  Integration  Roadmap,  Mr.  Adam  McCorkle,  GTRI 

•  Open  Systems  Architecture  (OSA)  and  Standard  Interfaces  as  Mission  Capability  Enablers,  William  H.  Mish,  Jr.,  AMSEC 

•  Naval  Air  Systems  Command  Integrated  In-Service  Reliability  Program  (IISRP),  Mr.  Les  Wetherington,  Integrated  In-Service  Reliability  Program  (IISRP) 
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Scm)ay,  October  23,  2005 

5:00  PM'7:00  PM  Registration  for  Tutorialsand  General  Conference 

(Tutorialsare  an  additional  $200  registration  fee) 


Monday,  October  24,  2005 

7:00  AM  '5  PM  Re<3 istra  tio n 

7  AM  Continental  Breakfast  for  Tutorial  AttendeesONLY 

(Tutorialsare  an  additional  $200  registration  fee) 


8:00  AM '5  PM 
12  Norm  '  1  PM 
1:00  PM '5  PM 
5:00  PM  '6  PM 


Tutorial  Tracks  (Please  referto  following  pagesforTutoria  Is  Schedule) 
Buffett  Lunch 

Tutorial  Tracks  (Please  referto  following  pagesforTutoria  Is  Schedule) 
Reception  in  Display  Area  (Open  to  All  Participants) 


Tuesday,  October  25, 2005 

7:00  AM  Registration  &  Continental  Breakfast 

8:15  AM  Introductions 

Mr.  Sam  Campagna,  Director,  Operations,  NDIA 

8:30 AM  Opening  Remarks 

Mr.  Bob  Rassa,  Director,  Systems  Supports  bility,  Raytheon; 

Chair,  Systems  Engineering  Division,  NDIA 

8:40  AM  -  9:30  AM  Keynote  Add  ness 

Mr.JohnLandon,  Deputy  Assistant  Secretary  of  Defense  (Nil) 

(C3ISR&  ITAcquisition) 

9:30  AM  '  10  AM  Brea  kin  Display  Area 

10:00  AM  '  12  Mwtpienary  Session:  Revitalization  of  Systems  Engineering  Within  DoD 

Moderator: 

Mr.  MarkSchaeffer,  Deputy  Director,  Defense  Systems,  and  Director, 

Systems  Engineering,  OUSD(A"f&L) 

Panelists: 

Mr.  Terry J aggers,  Director,  SAF/AQR  (Science,  Technology  &  Engineering) 

Mr.  C arlSel,  ASN  (RDA)CHENG 
Mr.  Doug  Wiltsie,  USArmy  (Invited) 

Mr.  Kelly  Miller,  NSA  (Invited) 

12  Norm, '  1:30  PM  Luncheon  Speaker 

Mr.  Greg  Shelton,  Vice  President,  Engineering  Manufacturing  Technology 
&  Quality,  Raytheon 

1:30  PM  '5PM  ConcurrentSessions(Please  referto  following  pagesforsession  schedule) 


5:00  PM  '  6:30  PM  Reception  in  Display  Area 


Monday,  October  24,  2005 


R&jlftratim  St  Cowtiiwevtal  Brenkfatt 

8:00  AM 

9:45  AM 

10:15  AM  12  HOOK  1:00  PM 

2:45  PM 

3:15  PM 

5  PM-SPM\ 

TRACK  1  How  to  Define  System 

Engineering  ProcessesThat are 


Tutorial 


Settlor  1A1 


Short  and  Usable 


Mr.  Tim  Olson,  Quality 


Improvement  Consultants,  Inc. 


TRACK 2 
Tutorial 


Integrating  Systems 
Engineering  with  Earned  Value 
Management 


Mr.  Paul  Solomon, 

Settlor  1A2  Northrop  Grumman  Corp. 


TP  AT  IT  3  Up-To-Date  Systems 
L  tLAUt \v  Requirements  Tutorial 

Tutorial 


Mr.  J  effrey  Grady , 

Settlor  IAS  JOG  Systems  Engineering,  Inc. 


TRACK  4  Exploring  the  System  Solution 
‘  Space  using  Behavior  Ana  lysis 


Settlor  IAS-  Mr.  James  bong,  VitechCotp. 


TP  Arts  c  Syste Softwa  re/ Ha  rd wa  re 
L  KAUK  o  Qua|ity  Assure  nee 

Tutorial 


Mr.  Al  Florence, 
Settlor  IAS  The  MITRE  Corp. 


TP  AC K  £  Innovative  Design  forSix Sigma 
IKAUKb  (DFSS)  Approachesto  Test  and 

Tutorial  Evaluation:  A  Hands-On 
Experience 


Dr.  Mark  Kiemele, 

Settlor  1AG  Air  Academy  Associates 


TKACK  7  object  Oriented  Systems 
Engineering  Methodology 

Tutorial  <00SEM) 


Dr.  Abraham  Meilich, 
Settlor  1A7  Lockheed  Martin 


Tt 


TRACK  8 
Tutorial 

Settlor  1A8 


TEA 


TRACK  1  How  to  Define  System 

Engineering  Processeslhat are 

Tutorial  Short  and  Usable  (Continued) 


Mr.  Tim  Olson,  Quality 

Settlor  1B1  improvement  Consultants,  Inc. 


TP  AC K  9  lnte  g  re  ting  Syste  ms 

Engineering  with  Earned  Value 


Tutorial 


Management  (Continued) 


Mr.  Paul  Solomon, 

Settlor  1B2  Northrop  Grumman  Corp. 


TRACKS  Up-To-Date  Systems 
Re  q  u  ire  me  nts  Tuto  ria  I 
Tutorial  (Continued) 


Mr.  J  effrey  Grady, 

Settlor  IBS  jog  Systems  Engineering,  Inc. 


TRACK  4  Exploring  the  System  Solution 
^  Space  using  Behavior  Ana  lysis 
TuTmA/il  and  Simulation:  Applying  M&S 
luroruu  tQ  System  Engineering  (Continued) 


Settlor  IBS- 


Mr.  James  bong,  V'rtech  Corp. 


'-rt>  at  iy  r  Systems/ Software/ Ha  id  ware  Qua  I- 
TKACK  S  ity  Assure  nee 

Tutorial  (Continued) 


Mr.  Al  Florence , 
Settlor  IBS  The  MITRE  Corp. 


TP  AC K  £  Innovative  Design  for  Six  Sigma 
J  <o  (DFSS)  Approachesto  Test  and 

T/v  W/,  1  Evaluation:  A  Hands-On  Experi- 
TutoruU  ence  (Continued) 


Dr.  Mark  Kiemele , 

Settlor  1BG  Air  Academy  Associates 


TRACK  7  objec1:  Oriented  Systems 
/  Engineering  Methodology 


Tutorial 


(OOSEMKContinued) 


Dr.  Abraham  Meilich, 
Settlor  1B7  Lockheed  Martin 


TKACK  8 
Tutorial 

Settlor  1B8 


7BA 


-rt>  a  ns  t  Systems  Engineering  Planning  - 

TKACK  7  ATutoria| 

Tutorial 


Col  Warren  Anderson,  OUSD 
Settlor  1C1  (ATM.)  Defense  Systems 


TKACK  2 
Tutorial 


Using  a  Measurement  Framework 
to  Successfully  Achieve  Measur¬ 
able  Results 


Mr.  Tim  Olson,  Quality 
Settlor  1C2  Improvement  Consultants 


'tv  Any  2  Requirements  Development  and 
LKACK  5  Management 

Tutorial 


Mr.  Al  Florence, 
Settlor  ICS  The  MITRE  Corp. 


TRACK  4 
Tutorial 


Air  Force  Integrated  Collabora¬ 
tive  Environment  (AF-IC  E)  -  An  Air 
Force  and  Industry  Partner 
overview  and  update 


Mr.  Rick  Peters, 

Settlor  ICS  Air  Force  Material  Command 


TP  ACK  C  "^e  Retum  on  Investment  from 
l  KAUIs.  v  Engineering  Best 

Tutorial  Practices:  An  Introduction 


Mr.  Thomas  Me  Gibbon, 
Settlor  ICS  nr  Industries 


TP  ACK  £  what  Makes  A  Simulation 
J  b  Credible?  Cost-Effective  W&A  in 

Tutorial  the  Systems  Engineering  Process 


Mr.  David  Hall,  SURVICE 
Settlor  ICG  Engineering  Company 


TKACK  7  obJect  Oriented  Systems 
Engineering  Methodology 
Tutorial  (OOSEM)(Continued) 


Dr.  Abraham  Meilich, 
Settlor  1C7  Lockheed  Martin 


TP  ACK  Si  Per,:o ^ability  (Performance  and 

1  V  Reliability)  Modeling 

Tutorial 


Dr.  Meng-Lai  Yin, 

Settlor  1C8  Raytheon 


TP  ACK  1  Systems  Engineering  Planning  - 
U4AUi\  i  A  Tutorial  (Continued) 

Tutorial 


Col  Warren  Anderson,  OUSD 
Settlor  1 VI  (ATSdJ  Defense  Systems 


TP  AC !4  9  Usin9  a  Measurement  Framework 
1  z  to  Successfully  Achieve 

Tutorial  Measurable  Results  (Continued) 


Mr.  Tim  Olson,  Quality 
Settlor  1V2  Improvement  Consultants 


tv  Any  2  Requirements  Development  and 
L  KACK.  5  Management 

Tutorial  (Continued) 


Mr.  Al  Florence, 
Settlor  1VS  The  MITRE  Corp. 


TKACK  4  AirForce  Integrated  Collabora- 
x  tive  Environment  (AF-IC  E)  -  An  Air 

Tutorial  Force  and  lndust[y  Pa[tner 

overview  and  update  (Continued) 


Mr.  Rick  Peters, 

Settlor  1VS  AirForce  Material  Command 

TP  ACK  C  "^e  ^etum  on  Investment  from 
L  KAUI\  S>  software  Engineering  Best  Prac- 

Tutorial  fees  An  Introduction 


Mr.  Thomas  Me  Gibbon, 
Settlor  1VS  fTT Industries 


TP  ACK  £  what  MakesA  Simulation  Cred- 
l  K AUK  o  jb|e?  Cost.Effective  VV&A  in  the 

Tutorial  ^ 


Mr.  David  Hall,  SURVICE 
Settlor  1VG  Engineering  Company 


TKACK  7  object  0riented  Systems 
Engineering  Methodology 
Tutorial  (OOSEMKContinued) 


Dr.  Abraham  Meilich, 
Settlor  1V7  Lockheed  Martin 


TP  AC  14  8  Performability  (Performance  and 
1  *  Reliability)  Modeling 

Tutorial 


Dr.  Meng-Lai  Yin, 

Settlor  1V8  Raytheon 
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3:00  PM\ 

3:30  PM 

TFACK  1 

The  Return  of  Disc  ip  line 

Technical  Planning  forAcquisition 

TFACK  1 

Implementation  of  Policy 

Systems  Engineering  Revitaliza¬ 

Engineering  for  Software 

Programs:  An  OSD  Perspective 

Requiring  Systems  Engineering 

tion  a  t  SPA  WA  R  Syste  ms  C  e  nte  r 

Assurance 

System*  Engtiieertiig 

System *  Engtiieertiig 

Plans  for  Air  Force  Programs 
-  Results  and  Implications 

Charleston 

Effedttimie** 

Effedbiuene** 

Session,  201 

Dr.  Yvette  Weber, 

Col  Wanen  Anderson, 

Session  2D1 

Mr.  Kevin  Kemper, 

Mr.  Michael Kutch,  Jr., 

Ms.  Kristen  Baldwin, 

HQAFMC,  LEAF 

OUSD  (ATEdJ  Defense  Systems 

US  Air  Force 

SPA  WAR  Systems  Center 

OUSD(ATSt) 

TRACK Z 

Technology  Readiness  Assessments:  A  Key 

Taxonomy  of  Operational  Risks 

TFACK2 

A  Method  for  Reasoning  About 

WBS  Based  Risk  Assessment 

Aspect  of  the  Systems 

an  Acquisition  Strategy 

System *  Engtiieertiig 

Engineering  Process 

System*  Engtiieertiig 

Effedtivme** 

Effedbivene** 

Session  202 

Dr.  Jay  Mandelbaum, 

Mr.  Brian  Gallagher, 

Session  2D 2 

Mr.  J  oseph  Elm, 

Mr.  Bruce  Heim, 

Institute  for  Defense  Analyses 

Software  Engineering  Institute 

Software  Engineering  Institute 

(DCMA)  Boeing  Long  Beach 

TRACK  3 

Applying  the  Systems  Engineering 

Intelligent  Data  Ana  lysis  Options  to  Support 

TFACK3 

Interweaving  Test  and  Evalu¬ 

Recent  Innovations  in  Design 

Flight  Testing  Airborne  Radar 

Approach  to  the  Test  and  Evaluation 

Aire  raft/ Ship  Systems  Testing 

ation  throughout  the  Systems 

forSix Sigma  (DFSS)  Testing 

Syste  ms  to  Improve  System 

Test  St  Evaluation  tit 
System *  Engtiieertiig 

Process 

Test  St  Evaluation  til 
System*  Engtiieertiig 

Engineering  Process 

Approachesto  Speed 
Technology  to  the 

Marketplace 

Performance 

Session  2C3> 

Mr.  Raymond  Beach, 

Mr.  Dean  Carico, 

Session  2D 3 

Mr.  Joseph  Tribble, 

Dr.  MaikKiemele, 

Mr.  Mark  London, 

NAVAIR 

Naval  Air  Warfare  Center 

A  VW  Tec  hnologies 

Air  Academy  Associates 

NAVAIR 

TRACK  4 

Guiding  DoD'smove  into  the 

Challenges  in  Development  of  System  of 

TFACK  4 

Real-Time  Tactical  Services  for 

Next  Generation  Enterprise 

Information  Age 

Systems  (SoS)  Arc  hitectures  in  a  Net  C  entric 

l 

the  GIG 

Information  Management 

Net  Centric  Operations 

Environment 

Net  Centric  Operation* 

Appliances 

Session  204 

Dr.  Abraham  Meilich, 

Session  2D4 

Mr.  John  Noble, 

Mr.  Michael  Lindow, 

Mr.  JackZavin,  ASD(NII)/DoD  CIO 

Lockheed  Martin 

£ 

J  HU  Applied  Physics  Laboratory 

The  MITRE  Coip. 

TFACK 5 

Intro  to  Logistics&  Sup  portability 

Condition  Based  Logistics 

TFACK  5 

FRACAS  Implementation  using 

Creating  a  Logistics  Health 

X*- 

ITLog 

Management  System 

Lo^itNcs 

Logvfbic* 

Session  205 

Mr.JenyBeck, 

Mr.  Ron  Wagner, 

V 

Session  2D 5 

Mr.  William  Jacobs, 

Mr.  Gary  O'Neill, 

OSD  Office  of  ADUSD(L&MR) 

CoBaLt  Technology 

Raytheon 

Georgia  Tech  Research  Inst 

TFACK  6 

Intro  to  Integrated  Diagnostics 

Diagnostic  Software  -  What  your  average 

§ 

TFACK  6 

Designing  for  Health;  A 

COTSBased  Solution  for 

developerdoesn't  know 

Methodology  for  Integrated 

Integrated  Test  and 

Integrated  Diagnostics 

Integrated  Diagnostic* 

Dia  g  no  Stic  sj  Prog  no  Stic  s 

Diagnostics 

Session  20G 

Mr.  Dennis  Hecht 

Mr.  Theodore  Marz,  Carnegie  Mellon  Uni¬ 

Session  2D6 

Mr.  Larry  Butler, 

Dr.  Ion  Neag, 

The  Boeing  Company 

versity  -  Software  Engineering 

Raytheon 

TYXCoip. 

TFACK  7 

System  Safety  in  Systems  Engineering  DAU 

System  Safety  in  the  Systems 

TFACK  7 

Revita  lizing  System  Safety  as 

Linking  System  Safety  to 

Integrating  MIL-STD-882 

Continuous  Learning  Module  Overview 

Engineering  Process 

One  of  the  Key  Elements  to 

Systems  Eng  ineering 

System*  Safety 

System  Safety 

Revitalizing  Systems  Engineer¬ 
ing  in  Department  of  Defense 
Acquisition  Programs 

Session  207 

Ms  Amanda  Za reeky, 

Booz  Allen  Hamilton 

Dr.  Ray  Terry, 

Session  2D7 

Col  Wanen  Anderson, 

Ms.  Paige  Ripani, 

Mr.  RickMilnarik, 

SURVICE  Engineering  Company 

OUSD  (ATSdJ  Defense  Systems 

Booz  Allen  Hamilton 

TFACK  8 

Proper  Specification  of  Software  Require¬ 

C-17  Software  Development  Process 

TFACK  8 

Successful  Verification  and 

Automated  Software  Testing 

Software  Sup  portability: 

ments 

Validation  Based  on  the  CMMI 

IncreasesTest  Quality  and 

A  Software  Engineering 

Software 

Software 

Model 

Coverage  Resulting  in  Improved 
Software  Reliability 

Perspective 

SuyportaJnUty 

SupportaJnUty 

Session  208 

Mr.  Al  Florence, 

Mr.  Hafez  Loiseyed i, 

Session  2D 8 

Mr.  Tim  Olson,  Quality 

Mr.  Frank  Salvatore,  High 

Mrs.  Stephany  Bellomo, 

The  MITRE  Corporation 

The  Boeing  Company 

Improvement  Consultants,  Inc. 

Performance  Technologies,  Inc. 

SAIC 
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8:15  AM 

TRACK  1 

Systems  Eigintierigg 
EffeAtivme** 

Tailorable  Decision  Ana  lysis  and  Resolution 
processand  too  Is  for  enterprise  wide 
application 

Defining  System  Development  Lifecycles 
to  Plan  and 

Manage  Projects  Effectively 

Session  3A1 

Mr.  Robert  Trifiletti,  Jr., 

USArmy  ARDEC 

Mr.  Bruce  Boyd, 

The  Boeing  Company 

TRACK  2 

Systems  Engineering 
Effe&iven*** 

Application  of  Risk 

Managementacross 

Engineering  and  Acquisition 

Requirements  Engineering  lipsand  Tricks 

Session  3A2 

Ms.  Rebecca  Cowerr-Hirsch, 

Defense  Systems  Agency 

Mr.  Frank  Salvatore, 

High  Performance  Technologies,  Inc. 

TRACKS 

Systems  Engineering 
Effectiveness 

Effective  SE  Metrics  Tailored  to  the  Acquisi¬ 
tion  Life  Cycle 

Innovative  Procurement  Strategies 

Session  3  A3 

Ms.  Laura  Troiola, 

USArmy  -  ARDEC 

Mr.  David  Eiband, 

Defense  Acquisition  University 

TRACK  4 

Net  Centric  Operations 

Session  3 At 

J  oint  Battle  Management  Command  & 
Control  RoadMap  -  Panel 

Moderators: 

Dr.  Vitalij  Garber,  Ms.  Robin  Quinlan,  DUSD 
(ATSdJ  DS/SI 

Panelists: 

Maj  Gen  C  ha  ties  Simpson,  USAF 

MG  Michael  Vane,  USA 

Joint  Battle  ManagementCommand  & 

Control  RoadMap  -  Panel 

Moderators: 

Dr.  Vitalij  Gather,  Ms.  Robin  Quinlan,  DUSD 
(ATEAJ  D& SI 

Panelists: 

Maj  Gen  C  ha  ties  Simpson,  USAF 

MG  Michael  Vane,  USA 

TRACK  5 

LogitUc* 

Improving  Sup  portability  on  Currently 
Deployed  Weapon  Systems 

Process  for  Evaluating  Logistics  Readiness 

Le  ve  Is  ( LRLs)  fo  r  A  c  q  u  isitio  n  Syste  ms 

Session  3A5 

Mr.  J  ohn  Sells, 

Tobyhanna  Army  Depot 

Mr.  Robert  Ernst 

NAVAIR 

TRACK  G 

Modeling  St 
Simulation 

Improving  M&S  Support  to  Acquisition 

Improving  M&S  Support  to  Acquisition 
(Continued) 

Session  3 AG 

Mr.  J  ames  Uollenbach, 

Simulation  Strategies,  Inc. 

Mr.  J  ames  Hollenbach, 

Simulation  Strategies,  Inc. 

TRACK  7 

System,  Safely 

A  Model  Linking  Safety,  "Threat  and  Other 
Critical  Causal  Factors  to  "Their  Mitig  a  tors" 
Relative  to  (Software,  Hardware,  and  Hu¬ 
man  System  Integration 

Mission  Sustainment  Through 

Acquisition  Environment,  Safety,  and 
Occupational  Health  (ESOH)  Risk 

Management 

Session  3A7 

Ms.  Janet  Gill, 

NAVAIR 

Ms.  Karen  Gill, 

Booz  Allen  Hamilton 

TRACK  8 

Software 

SuyportaTiUty 

Sustaining  Software-Intensive  Systems- A 
Conundrum 

Algorithm  Description 

Documentation  and  Validation  Process 

Session  3 AS 

Ms.  Mary  Ann  Lap  ham, 

SEI 

Mr.  Michael  K.  Bailey, 

Raytheon 
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TRACK  1 

System*  Engineering 
EffeTtumie** 

System  Engineering,  Program  Manage-  Tailoring  USAF  Systems 

ment conjoined  Disciplines overthe  Project  Engineering  forthe  Life  Cycle:  One  Shape, 

LifeCycle  Multiple 

Dimensions 

Session  3B1 

Mr.  ]/\/illiam  Lyders, 

ASSETT,  Inc. 

Mr.  Jeff  Loren, 

MIC  Technologies,  Inc.  (SAF/AQRE) 

TRACK  2 

System*  Engineering 
EffeCtUmtes* 

Engineering  and  Implementing  RMS  Engi¬ 
neering  D1C  Metrics 

System  Engineering  Metrics 

Se**ion  3B2 

Mr.  Edward  Casey, 

Raytheon  Missile  Systems 

Mr.  J  ames  Miller, 

United  States  Air  Force 

TRACK  3 

System *  Engineering 
EffeCtiveres* 

Using  Systems  Engineering 

Principles  to  Transform  R  &  D  Into  a  Military 
System  Solution 

Next  Generation  Combat  Systems-  An 

Overview  of  Key  Development  Concepts 

Session  3B3 

Dr.  James  Dill, 

Foster-Miller 

Mr.  Matthew  Montoya, 

The  J  HU  Applied  Physics  Laboratory 

TRACK  4 

Net  Centric  Operation* 

Network-Centric  Capabilities 

Development  for  Ground  Mobile  Forces 

Testing  Net-Centric  Systems  of 

Systems:  Applying  Lessons  Learned  from 

Distributed  Simulation 

Session  3B4 

Ms.  Diane  Hanf, 

The  MITRE  Corp. 

Mr.  R.  Douglas  Flournoy, 

TRACK  5 

LogidUc* 

The  Management  of  Log istic s  in  Large 

Scale  Inventory  Syste  ms  to  Support 
Weapon  System  Maintenance 

System  of  Systems  Ana  lysis  of  Future 

Combat  System  Sustainment  Requirements 

Session  3B5 

Mr.  Eugene  Beardslee, 

SAIC 

Mr.  Ivan  Wolnek, 

The  Boeing  Company 

TRACK  G 

Modeling  St 
Simulation 

Next  Generation  Manufacturing  Tech¬ 
nology  Initiative  and  the  Model-Based 
Enterprise 

Problem  Space  Modeling 

Session  3BG 

Mr.  Richard  Neal, 

IMT1 

Mr.  J  effrey  O.  Grady, 

JOG  Systems  Engineering,  Inc. 

TRACK  7 

System  Safely 

Session  3B7 

Army  Acquisition  Programs' 

Installations,  Environmental,  Safety,  and 
Occupational  Health 

Considerations 

Mr.  Donald  Artis,  Jr.,  Office  of  the 
DASA(ESOH) 

Current  DoD  Acquisition  Policiesand 

Guidance  on  the  use  of  M IL-STD-882D  to 

Integrate  Environment,  Safety,  and  Occu¬ 
pational  Health  (ESOH)  Considerations  into 
the  Systems  Engineering  Process 

Mr.  Sherman  Forbes, 

USAF-  SAF/AQRE 

TRACK  8 

Legaxy  System* 

Siestatnment 

"The  Integration  of  Systems  Engineering  and 
Enterprise  Architecture  with  respect  to  the 
Modernization  of  Legacy  Systems-  Panel 

The  Integration  of  Systems  Engineering  and 

Enterprise  Architecture  with  respect  to  the 
Modernization  of  Legacy  Systems-  Panel 
(Continued) 

Session  3B8 

Mr.  Owen  Williams,  Science 

Applications  International  Corp. 

Mr.  Owen  Williams,  Science 

Applications  International  Corp. 
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TRACK  1 

Systems  Eng  uttering 
EffeAbUmtess 

Arc hitec tune  Based  Systems 

Engineering  And  Integration 

A  Complementary  Approach  to  Enterprise 

Syste  msEngineering 

Session  3C1 

Dr.  RickHabayeb,  Virginia  Tech 

Dr.  Brian  White,  The  MITRE  Corp. 

TRACK  2 

Technical  Performance  Measures 

Turbo  Tax  for  Syste  ms  Engineering 

Systems  Engineering 
Effectiveness 

Session  3C2 

Mr.  J  im  Oakes, 

BAE  Systems 

Mr.  Michael  Kutch,  J  r., 

SPAWAR 

TRACK  3 

Systems  Engineering 
Effectiveness 

Converting  High-Level  Systems 

Engineering  Policy  to  a  Workable  Program 

Revitalization  of  Systems  Engineering;  Past, 
Present  and  Future 

Session  3C3 

Mr.  James  Miller, 

US  Air  Force 

Ms.  Karen  Bausman, 

USAF  Center  for  Systems  Engineering 

TRACK  4 

Net  Centric  Operations 

What  is  the  difference  between 

Multi-Level  Security  (MLS)  and  Multiple 
Secure  Levels  (MSL)  Architectures  and  why 
do  you  care? 

A  Network  Centric  Warfare  Platform  With 

Multiple  Missions  in  Mind 

Session  3C4 

Mr.  Paul  Vazquez,  J  r., 

Raytheon  NCS 

Mr.  Pederjurrgck, 

CloudShield  Technologies 

TRACK  5 

LogidUcs 

Reaping  the  benefits  of  PBL/CSL 

Priming  &  Tuning  the  ERP/MRO 

Engine:  Integrated  Through-life 

Sup  portability  Data  Management 

Session  3C5 

Ms.  Denise  Duncan, 

LMI 

Mr.  Patrick  Read, 

Pennant  Canada,  Ud 

TRACK  6 

Update  on  SysML 

Data  Management  to  support  M&S 

Modeling  St 
Sinudation, 

Session  3C6 

Mr.  Rick  Steiner, 

Raytheon 

Ms.  Denise  Duncan, 

LMI 

TRACK  7 

System  Safety 

Lessons  Learned  with  the  Application  of 

MIL-STD-882D  Within  the  Navy's  Weapon 
System  Explosives  Safety  Review  Board 

Industry  perspectives  and  identified  barriers 

to  the  use  of  M IL-STD-882D  for  integrating 

ESO  H  c  o  nsid  e  ra  tio  ns  into  Syste  ms 

Session  3C7 

Ms.  Mary  Caro, 

Naval  Ordnance  Safety  &  Security  Activity 

Mr.  J  on  Derickson, 

United  Defense 

TRACK  8 

Legacy  System* 

Snsta/nment 

The  Aging  Transport  Systems 

Rulemaking  Advisory  Committee:  Back¬ 
ground,  Results  and  Future  Impact  on  the 
Aviation  Industry 

J  a mmer  Integration  Roadmap 

Session  3C8 

Mr.  KerrtHollinger, 

The  MITRE  Corp. 

Mr.  Adam  McCorkle, 

Georgia  Tech  Research  Institute 

3:30  PM 


Tt 
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TRACK  1 

Systems  Engineering 
Effectiveness 

Implementing  SE  Processesto 
Balance  Cost  and  Technical 
Performance 

A  Revolutionary  Model  to  Sup¬ 
port  Early  CAIV  Tradesand  Cost 
Predictions 

Session.  3V1 

Dr.  Mary  Anne  Herndon,  SAIC 

Mr.  Bryan  Piggott,  InfoEdge 

TRACK  2 

Systems  Engineering 
Effectiveness 

A  Practical  Application  of  the 
Non-Advocate  Review 

Systems  Engineering  and  the 
Software  Laws  of 
Thermodynamics 

Unmanned  Aerial  Vehicle 
Survivability  Influence  on 

System  Life  Cycle  Cost 

Session  3V2 

Mr.  Bruce  Nishime, 

The  Boeing  Company 

Dr.  Thomas  Christian,  Jr., 
402SMXG 

Mr.  CharlesPedriani, 

SURVICE  Engineering 

TRACK  3 

Systems  Engineering 
Effectiveness 

AFRL Systems  Engineering  System  Engineered  Research 

Initiative  -  Risk  Management  for  and  Development 

Science  and  Technology  Management 

Session  3V3 

Mr.  William  Notts, 

USAF-AFRL 

Dr.  Steven  Ligon, 

SAIC 

TRACK  4 

Net  Centric  Operations 

Systems  Engineering  Analysis 

and  Control  Methodsto  Assure 
Electromagnetic  Spectrum 
Access 

A  Strategy  for  Managing  the 

Development  and  Certification 
of  Net-Centric  Services  within 
the  Global  Information  Grid 

Session  3V4 

Mrs.  Rerrae  Carter, 

DISA  Defense  Spectrum  Office 

Mr.  Bernal  Allen, 

Defense  Systems  Agency 

TRACK  5 

Best  Brattices  St 

StantJar, donation 

On  the  Shoulders  of  CMM:  CMMI  for  Services 

CMMI  +C015+0A  +nNIH  =less 
(cost)  +more  (capability) 

Session  3  V5 

Mr.  Luke  Campbell, 

NAVAIR 

Mr.JuanCeva, 

Raytheon  RIS 

TRACK  6 

Modeling  St 
Sinudation 

Enterprise  Digital  Data 
Management 

The  Use  of  Simulation  in  the 
Management  of  Logistics  in 
Large  Scale  Inventory  Systems 
to  Support  Weapon  System 
Maintenance 

Ensuring  Accomplishment  of 
Performance  Based  Logistics 
Objectives  Using 

Model-Based  Systems  Engineer¬ 
ing 

Session  3V6 

Ms.  Cynthia  Hauer,  Millennium 
Data  Management;  Inc. 

Mr.  Eugene  Beardslee, 

SAIC 

Mr.  Timothy  Tritsch, 

V'rtech  Corp. 

TRACK  7 

System  Safety 

Comparisons  and  Contrasts 

Between  ISO  14001,  OHSAS 
18001,  and  M IL-STD-882D  a nd 
their  Suitability  for  the  Systems 
Engineering  Process 

Evo  lutio  n  of  M  ilita  ry  Sta  nd  a  rd 

882E 

USMC  Expeditionary  Fight¬ 

ing  Vehicle  (EFV):  A  Vehicle 
Designed  with  Environmental, 
System  Safety,  and  Occupa¬ 
tional  Health  (ESOH)  in  Mind 

Session  3V7 

Mr.  Kenneth  Dormer,  USAF 
Contractor  (SAF/AQ  RE) 

Mr.  J  immy  Turner, 

Raytheon 

Ms.  Sandra  Fenwick, 

USMC  DRPM  AAA 

TRACK  8 

Legacy  Systems/ 

Open  Systems 

NAVAIR  Integrated  In-Service 
Reliability  Program  -  Aging  Air- 
craftyKeeping  Legacy  Systems 
Viable 

Delivering  Effective  Solutions 
in  the  Age  of  Open  Source 
Technology 

Session  3V8 

Ms.  Debbie  Vergos, 

Naval  Air  Systems  Command 

Mr.  Edward  Beck, 

Computer  Sciences  Corp. 

Conference  Adjmuvn  for  the  Day 
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TRACK  1 

Systems  Engineering 
Ejfe&Umiest 

A  Systems  Afford  ability  Approach 

Using  Raytheon  Six  Sigma  Design 

Requirements  Engineering  Tips  and  Tricks 

Session  4A1 

Ms.  Yvette  Thornton, 

Raytheon 

Mr.  Frank  Salvatore, 

HPli 

TRACK  2 

Systems  Engineering 
EjfedbUmiess 

Surveying  SE  Effectiveness 

Integrated  Survivability  Assessment  (ISA)  in 
the  Systems  Engineering  Process 

Session  4A2 

Mr.  J  oseph  Elm, 

Software  Engineering  Institute 

Mr.  David  H.  Hall, 

SURVICE  Engineering  Company 

TRACK  3 

Systems  Engineering 
Ejfedbumtess 

10  Golden  QuestionsforConcept  Explora¬ 
tion  and  Development 

The  C -17  Systems  Engineering 

Experience 

Session  4. A3 

Dr.  Dan  Surber, 

Raytheon  Technical  Services  Co. 

Mr.  Kenneth  Sanger, 

The  Boeing  Company 

TRACK  4 

Net  Centric  Operation * 

Net  Centric  Test&  Evaluation 

Profiling  and  Testing  Procedures  for  a  Net- 
Centric  Data  Provider 

Session  4A4 

Mr.  Ric  Harrison, 

DISA 

Mr.  DerikPack,  Space  &  Naval  Warfare 

Systems  Center-  Charleston 

TRACK  5 

Best  Er&ctices  St 

Stati) an, dotation 

Process  Architecture  and  Criteria  for  Les¬ 
sons  Learned 

Successful  StrategiesTo  Improve  Your 
Requirements 

Session  4A5 

Mr.  Thomas  Cowles, 

Raytheon  Space  &  Airborne  Systems 

Mr.  Tim  Olson, 

Quality  Improvement  Consultants,  Inc. 

TRACK  6 

Modeling  St 
Simulation 

Application  of  a  State-Machine  Model  for 
the  Ana  lysis  &  Optimization  of  Task-Post- 
Process-Use  pPPU]  and  Task,  Process, 
Exploitation  and  Disseminate  [1PED] 
Processes 

A  Heuristic s Systems  Engineering  Approach 
to  Modeling  and  Ana  lysis  of  the  U.S.  Strate¬ 
gic  Highway  Network  (STRA  hi  NET) 

Session  4AG 

Mr.  Ric  hand  Sorensen, 

V'rtech  Corp. 

Mr.  Gerard  Ibana, 

Southern  Methodist  University 

TRACK  7 

Education  St  Training 

Educating  Future  Systems  Engineers:  USMili 
tary  Academy  Reception-Day  Simulation 
and  Optimization 

in  SE 

Session  4A7 

LTC  Simon  Goetger, 

Department  of  Systems  Engineering 

TRACK  8 

Net  Centric  Operation s 

The  Role  of  the  Opera  torand  System 
Engineerin  the  Force  Modernization 
Environment 

TBA 

Session  4A8 

Mr.  Thomas  Nelson, 

J  acobs  Sverdrup 

Si  CcmMjj&wtal  Brenkfa^t 
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TRACK  1 

System*  Engineering 
Effe&ivenes* 

Session  4B1 

How  the  Pro-Active  Program  (Project) 
Managerusesa  Systems  Engineer's  Trade 
Study  asa  Management  Tool,  and  not  just 
a  Decision-Making  Process 

Mr.  Art  Felix, 

US  Navy 

Experience  in  Supporting  Systems  Engineer¬ 
ing  Project  Management  Using  CORE 

Mr.  George  Blaine, 

United  Dfense,  LP 

TRACK  2 

A  systems  approach  to  Accelerating  Test¬ 

Applying  the  Systems  Engineering  Method 

ing,  a  case  study 

to  the  Joint  Capabilities 

System s  Engineering 
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Abstract 


Raytheon 


•  Complicated  algorithms  difficult  to  translate  to  SRS 

-  Highlights  the  division  between  algorithm  validation  and  software 
verification 

-  Results  in  a  disjoint  agreement  between  Systems  and  Software 
engineering 

•  Algorithm  Description  Document 

-  Documents  the  life-cycle  of  algorithms 

-  Includes  trade  study  analyses  and  validation  results  to  illustrate  details 
design  intent 

-  Allows  for  a  mutual  engineering  understanding 

-  Describes  the  most  recent  design 

•ADD  Process  is  presented 

ADD  defines  the  difference  between 
Verification  and  Validation 


The  contents  of  this  material  reflect  the  views  of  the  authors.  Neither  the  Federal  Aviation  Administration  nor  the  Department  of  Transportation  make  any  warranty  or 
guarantee,  or  promise,  expressed  or  implied,  concerning  the  content  or  accuracy  of  the  views  expressed  herein. 
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Navigation  and  Landing  Systems 
Developments 
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•  Raytheon’s  NLS  group  developed  ADDs  and  simulation  tools 
to  test  algorithms 

-  Process  successfully  implemented  during  initial  operating  phase  of  an 
SBAS  approved  for  use  in  commercial  aviation 

-  Customer  and  Raytheon  currently  planning  upgrades  to  improve 
service  and  availability 

-  A  combination  of  algorithm  enhancements  required  to  achieve  future 
operational  goals 

To  be  rolled  out  sequentially  over  the  next  few  years 

•  Process  expanded  to  include  all  safety  algorithms  in  NLS 
program  subsystems 

-  Continues  to  be  used  by  all  SBAS  programs  within  Raytheon’s  NLS 
group  as  the  primary  algorithm  development  tool 


ADD  allows  Software  Enhancements 


The  contents  of  this  material  reflect  the  views  of  the  authors.  Neither  the  Federal  Aviation  Administration  nor  the  Department  of  Transportation  make  any  warranty  or 
guarantee,  or  promise,  expressed  or  implied,  concerning  the  content  or  accuracy  of  the  views  expressed  herein. 
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Software  Development  Challenges 
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•  Typical  programs  include  safety-of-life  systems  centered  on 
math-intensive  algorithms 

-  New  insights  in  antenna  design,  ionospheric  behavior  and  error 
mitigation  lead  to  algorithm  redesigns 

•  The  safety-of-life  requirement  suggests  the  use  of  an 
RTC/DO-178B  Level  B  process 

•  NLS  programs  consist  of  software  developed  to  DO-1 78B 
Level  B  and  Level  D  standards 

-  Level  B  software  passes  through  rigorous  set  of  design  and  testing 
requirements 

-  Level  B  coding  involves  creating  a  formal  SRS,  ensuring  that  all 
requirements  in  the  SRS  are  addressed  in  the  code,  and  formally 
testing  all  branches  of  the  code  for  conformity 

Ensure  software  meets  safety  requirements 


The  contents  of  this  material  reflect  the  views  of  the  authors.  Neither  the  Federal  Aviation  Administration  nor  the  Department  of  Transportation  make  any  warranty  or 
guarantee,  or  promise,  expressed  or  implied,  concerning  the  content  or  accuracy  of  the  views  expressed  herein. 
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Software  Requirements  Document 
Considerations 
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•  SRS  is  relatively  expensive 

-  Each  idea  broken  down  into  modules  used  to  generate  pseudo  code 

-  Common  for  SRS  updates  to  lag 

•  SRS  focuses  on  how  code  is  supposed  to  operate 

-  Does  not  capture  discussions  and  trades  that  justify  the  algorithms 

-  Various  filters  and  algorithms  in  the  system  all  require  analysis 

•  Scientists  operate  in  a  results-based  paradigm 

-  More  effort  is  concentrated  on  the  results  of  the  algorithms  to  be  correct 
than  that  pseudo  code  to  be  clear 

-  As  a  consequence,  system  prototypes  were  correct  implementations  of 
the  SRS  but  not  correct  implementations  of  the  algorithms  envisioned 

-  Software  may  have  been  appropriately  verified,  but  was  not  a 
guaranteed  implementation  of  validated  algorithm 


Methodology  concentrates  on  Algorithm  Validation 

The  contents  of  this  material  reflect  the  views  of  the  authors.  Neither  the  Federal  Aviation  Administration  nor  the  Department  of  Transportation  make  any  warranty  or 
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System  Engineering  Methodology 

•  Design,  prototype,  tune  and  validate 

•  Process  has  3  goals 

-  Allow  flexibility  to  rework  algorithms  and  code 

-  Capture  information  that  describes  how  decisions  were  made 

-  Focus  on  ideas  and  results,  not  implementation  details 


Algorithms  are  proposed  and  described  in  ADD 

The  contents  of  this  material  reflect  the  views  of  the  authors.  Neither  the  Federal  Aviation  Administration  nor  the  Department  of  Transportation  make  any  warranty  or 
guarantee,  or  promise,  expressed  or  implied,  concerning  the  content  or  accuracy  of  the  views  expressed  herein. 
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Algorithm  Generation 
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•  Offline  studies 

-Trading  one  approach  against  another  or  examining  historical  data 

-  Included  in  the  ADD  to  give  an  understanding  for  the  motivation  of  the 
algorithm 

•  Design  is  prototyped  into  a  simulation  of  deliverable  system 

-  If  the  design  has  not  been  fully  decided,  the  prototype  engineer  will  use 
best  judgment  to  get  a  version  working 

-  If  there  are  multiple  competing  designs,  they  are  coded  under  compile 
flags  and  comparison  simulations  are  executed 


ADD  includes  off-line  studies 


The  contents  of  this  material  reflect  the  views  of  the  authors.  Neither  the  Federal  Aviation  Administration  nor  the  Department  of  Transportation  make  any  warranty  or 
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Algorithm  Validation 
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•  To  validate  that  code  operates  as  expected  the  simulation  is 
executed 

-  Code  has  extensive  debugging  capabilities  to  generate  reports  of 
algorithm  functions 

-  Compared  to  the  algorithm  description  in  the  ADD  to  determine  if  it  has 
the  correct  behavior 

•  Tuning  effort  where  the  algorithm  is  optimized 

-  Algorithm  is  modified  in  the  simulation  until  it  meets  expectation  results 

-  Summarized  in  ADD 

•  Once  tuned,  algorithm  design  described  in  ADD  is  updated 

•  ADD  reviewed  to  ensure  that  the  results  are  correct  and  that 
all  anomalies  are  explained 

•  ADD  given  to  Software  Engineering  for  implementation 

All  validation  efforts  summarized  in  ADD 


The  contents  of  this  material  reflect  the  views  of  the  authors.  Neither  the  Federal  Aviation  Administration  nor  the  Department  of  Transportation  make  any  warranty  or 
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Software  Engineering  Methodology 
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•  Generate  software  requirements,  code  and  verify 

•  Process  has  3  goals 

-  Ensure  one-to-one  correspondence  between  specification  and  code 

No  unimplemented  requirements  and 

No  code  that  is  not  described  in  the  requirements 

-  Ensure  that  the  software  has  been  coded  to  meet  the  prescribed 
RTC/DO-178B  safety  level 

All  branches  tested  for  correctness  and  robustness 


Methodology  concentrates  on  algorithm  validation 

The  contents  of  this  material  reflect  the  views  of  the  authors.  Neither  the  Federal  Aviation  Administration  nor  the  Department  of  Transportation  make  any  warranty  or 
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Algorithm  Implementation 
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•  Software  Engineering  updates  the  SRS  with  requirements 
generated  from  ADD 

•  Design  review  held  to  ensure  the  algorithm  described  in  the 
ADD  can  be  implemented  in  a  manner  consistent  with  Level 
B  design 

-  It  must  be  possible  to  prove  that  the  code  will  operate  consistently  in  a 
manner  as  described  in  the  SRS 

-  System  operational  software  is  coded  from  the  requirements 

If  the  deliverable  code  is  being  generated  by  perfecting  simulation  code,  the 
Software  Engineer  must  ensure  that  the  SRS,  and  not  the  prototype,  is  used 
as  the  source  of  requirements. 

-  The  operational  code  is  to  be  free  of  dead  or  unused  code,  debugging 
code  and  any  version  of  the  algorithm  other  than  the  final  version 
described  in  the  SRS 


SRS  updated  from  ADD 


The  contents  of  this  material  reflect  the  views  of  the  authors.  Neither  the  Federal  Aviation  Administration  nor  the  Department  of  Transportation  make  any  warranty  or 
guarantee,  or  promise,  expressed  or  implied,  concerning  the  content  or  accuracy  of  the  views  expressed  herein. 
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RTC/DO-178B  Level  B  Coding 


Raytheon 


•  Level  B  software  is  coded  in  a  style  that  prohibits  unsafe 
programming  constructions 

•  Software  Engineers  specially  trained  in  Level  B  programming 
techniques 

•  Once  complete,  the  simulation  code  is  available  as  a 
resource  for  comparison  testing 

•  Discrepancy  between  deliverable  and  prototype  code  is 
comparatively  great 

-  Prototype  coders  are  engineers  and  mathematicians  focused  on 
validation  and  benefit  exclusively  from  executing  the  simulation 

-  Prototype  not  developed  to  meet  any  specific  coding  standards 


Operational  code  required  to  pass  though  rigorous 
design  and  testing  requirements 


The  contents  of  this  material  reflect  the  views  of  the  authors.  Neither  the  Federal  Aviation  Administration  nor  the  Department  of  Transportation  make  any  warranty  or 
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Limitations 
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•  There  is  risk  in  overestimating  the  code  correctness  of  the 
validated  algorithms  in  the  simulation 

-  The  simulation  is  validated  by  checking  that  its  algorithms  function  as 
expected 

-  It  is  possible  to  have  simulation  code  that  gives  correct  or  nearly  correct 
results  while  still  containing  coding  errors 

•  If  integration-level  testing  of  the  operational  software  is  less 
rigorous  because  of  confidence  in  the  algorithm  validation  of 
simulation  code,  some  corner  robust  cases  may  be  under 
tested 


Robust  cases  may  be  under  tested 


The  contents  of  this  material  reflect  the  views  of  the  authors.  Neither  the  Federal  Aviation  Administration  nor  the  Department  of  Transportation  make  any  warranty  or 
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Algorithm  Changes 


Raytheon 


•  ADD  under  control  of  Algorithm  Design  Team 

•  SRS  under  control  of  Software  Engineers 

-  ADD  and  SRS  kept  synchronized  to  facilitate  communication  between 
the  two  groups 

-  In  this  way  a  Software  Engineer  is  unable  to  originate  a  change  to  an 
algorithm  that  is  not  reviewed  by  the  algorithm  designer 

-  Similarly,  the  analyst  is  unable  to  introduce  a  subtle  change  in  the 
algorithm  that  is  not  captured  in  the  code 

•  Once  the  algorithms  themselves  are  defined,  the  algorithms 

and  related  information  are  documented  in  the 


ADD  is  approved  by  System  Engineering 

The  contents  of  this  material  reflect  the  views  of  the  authors.  Neither  the  Federal  Aviation  Administration  nor  the  Department  of  Transportation  make  any  warranty  or 
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Change  Process 


Changes  are  made  and  approved 
by  a  change  review  board 

The  contents  of  this  material  reflect  the  views  of  the  authors.  Neither  the  Federal  Aviation  Administration  nor  the  Department  of  Transportation  make  any  warranty  or 
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Conclusion 
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•  ADDs  and  a  validation  process  has  helped  programs  to  be 
more  agile  in  the  face  of  rapid  algorithm  redesign 

•  By  centralizing  information  on  algorithm  tuning  and 
validation,  it  is  easier  to  understand  the  history  and 
justification  of  decisions  made 

•  By  formatting  the  information  to  be  more  accessible  to  the 
engineers  and  scientists,  it  has  kept  them  more  engaged  in 
the  process  of  document  review 


ADDs  have  led  to  a  safer  and  more  correct  product 

The  contents  of  this  material  reflect  the  views  of  the  authors.  Neither  the  Federal  Aviation  Administration  nor  the  Department  of  Transportation  make  any  warranty  or 
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Briefing  Overview 


■  Problem  Definition  and  Scope 

■  Background 

■  Challenges 

■  Solving  the  problem 


Problem  Definition 


■  Classic  inventory  problems 

■  How  much  to  stock? 

■  When  to  order? 

■  How  much  to  order? 

■  Difficult  problem  without  demand  data 

■  When  will  the  stock  location  run  out  of  inventory? 

■  Multiple  Consumers,  Multiple  interrelated 
maintenance  tasks 

■  No  access  to  planning  and  production  data 

■  Multi  pie- criteria  optimization  problem 


Minimize  I  nventory 
Management  Cost 


■  Holding  Cost 

■  Parts 

■  Physical  Space 

■  Stocking  Labor 

■  Penalty  Cost 

■  I  die  workers 

■  Delayed  tasks 


■  Order  Cost 

■  I  nventory  Review 

■  Shipping 

■  Receiving 

■  Forward  Locating 


Maximize  Performance 


Inventory  Bin  Fill  Rate 
Service  Rate 
I  nverse  Waiting  time 


Background 

■  Bench  stock  inventory 

■  More  than  400,000  stock  locations 

■  More  than  100,000  inventory  items 

■  Over  1.8  million  transactions  over  three  years 

■  Multiple  weapon  systems:  F- 15,  F-16,  C-130, 
C- 141,  C-5,  KC- 135,  B-1B,  B-52,  B-2,  E-3,  E- 
6A,  FA- 18,  P-3,  H-60,  AV-8,  and  others. 

■  Multiple  Sites:  OCALC,  OOALC,  WRALC, 

J  XNADEP,  Nl  NADEP,  CPNADEP 


Challenges 

■  Detecting  stock-out  conditions 

■  Forecasting  demand 

■  Identifying  inventory  policy  errors 

■  Compensating  for  variability  of  the 
maintenance  environment 

■  No  access  to  planning  and  production 
data 


More  Challenges 


■  Exact  item  count  in  inventory  location  is 
unknown 

■  Accurate  inventory  review  is  not 
economically  feasible 

■  Reorder  level  is  an  estimate 

■  Reorder  quantity  is  fixed 

■  I  terns  removed  from  the  stock  locations 
for  direct  use 


Insights:  Inventory 
Segmentation 


Low  value  parts  with  highly  variable 
and  high  volume  demand  can  have 
high  reorder  thresholds  (safety 
stock)  to  avoid  stockouts  without 
incurring  excessive  inventory  costs. 

High  value  parts  with  highly  variable 
and  high  volume  demand  would 
benefit  from  demand  forecasts  based 
on  anticipated  needs  to  minimize  the 
value  of  stock  on  hand. 

Low  value  parts  with  low  variability 
and  high  volume  demand  could  be 
“auto-replenished”  based  on  historical 

demand  patterns  -  lowering  inventory 
review  costs. 


High  value  parts  with  low  variability 
and  high  volume  demand  could  be 
forecasted  to  assist  suppliers,  but 

should  be  reviewed  more  frequently 

to  minimize  overstock  conditions. 


Stock  Out  T riggers 


Surge  in 
requirements 
Increasing 
trend 


New  Part 


Unanticipated 

Demand 


Stock  Out 


Solving  the  problem:  Daily 

■  Unmanned  Bench  Stock  Location 

■  Material  is  placed  in  a  bin  and  mechanic  takes  what  he  needs 

■  If  a  bin  is  empty,  inventory  manager  is  notified  and  generates  an 
emergency  PR  to  Vendor 

■  Conduct  physical  review  of  each  bench  stock  location  twice  per 
week  to  create  routine  replenishment 

■  Emergency  Requirements  Management 

■  Each  emergency  requirement  is  tracked  from  birth  to  death 

■  Stock  outage  Management 

■  Intensive  management  to  ensure  parts  are  in  the  bin 

■  Focuses  manager  on  potential  problems 

■  Web  based  asset  visibility 

■  EDI  Ordering  and  Invoicing 


Enterprise  Supply  Chain 
System:  SCOPTIMA™ 

■  Oracle  DBMS  with  custom  user  and  system  interfaces 

■  User-friendly  interface  for  management  queries 

■  Maintains  record  of  supply  chain  events 

■  Bin  scan  to  generate  potential  order  (hand-held  device) 

■  Order  generation  based  on  rule  set  (automated) 

■  EDI  order  placement  to  vendor(s) 

■  Order  receipt  and  replenishment  (physical  inspection  and 
confirmation) 

■  EDI  invoicing 

■  Supports  analysis  to  identify  historical  demand 
patterns  and  relationships  through  simulation  and 
data  mining 
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Applying  Technology 


■  Data  Warehousing 

■  Data  Mining 

■  Simulation 

■  Time- series  Forecast  modeling 

■  Dynamic  data  display 
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Supply  chain  segment  view 


SCOPTI MA™  Data  Mining 


I  ntegration  of  SLAM  and 
SCOPTI MA™ 


■  Automatically  run  simulation  of  Bin  Q,r 
strategy  to  verify  results 

■  Support  SCOPTI  MA  decision  making 
using  simulation 

■  Verification  of  Q,r  changes  when  bin  agent 
determines  need  for  change 

■  "Agent"  based  approach 

■  Automated 


I  integration 


■  Spawn  process  from  SCOPTI MA  to 
invoke  AweSI  M  model 

■  I  nterface  via  Data  Files  or  Database 

■  Results  returned  to  SCOPTI  MA  via  Data 
Files  or  Database 


Architecture 


AweSIM  Database: 
General  Bin  Model 
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Conclusion 


Challenging  Supply  Chain  Problems 
Complex  inventory  model 
I  nteg rated  Enterprise  System 
Advanced  technology  solutions 
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The  Use  of  Simulation  in  the 
Management  of  Logistics  in  Large 
Scale  I  nventory  Systems  to  Support 
Weapon  System  Maintenance 
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Hank  Grant,  U  of  Oklahoma 


Summary 


■  400,000  bins  of  material 

■  Multiple  Sites 

■  Varying  part  size 

■  Rivets  to  wings 

■  Difficult  to  monitor  and  forecast 

■  No  "production"  plan 

■  Demand  forecast  based  on  approximation  of 
inventory  level 

■  Empty  -  partial  -  full 


Summary 


■  General  autonomous  bin  simulation 

■  Agent  based  approach 

■  Data  (File)  Driven 

■  I  nterface  with  external  software 

■  Bins  simulated  over  50  years  of 
operation 


Summary 


■  Underlying  Simulation  Technology 


■  VI:  GPSS  model 
-  V2:  SLAM/AWESI M 


■  Simulation  technology  is  transparent  to 
the  user 


I  ntegration  of  SLAM  and 
SCOPTIMA™ 

■  Automatically  run  simulation  of  Bin  Q,r 
strategy  to  verify  results 

■  Support  SCOPTI MA  decision  making 
using  simulation 

■  Verification  of  Q,r  changes  when  bin  agent 
determines  need  for  change 

■  "Agent"  based  approach 

■  Automated 

J»  m 


I  integration 


■  Spawn  process  from  SCOPTI MA  to 
invoke  AweSI  M  model 


I  nterface  via  Data  Files  or  Database 


■  Results  returned  to  SCOPTI  MA  via  Data 
Files  or  Database 


Architecture 


AweSIM  Database: 
General  Bin  Model 


Summary  of  Simulator 
Capabilities 


■  Individual  Bins 

■  Multiple  Bins 

■  Sensitivity  Analysis 

■  Demand  Rate 

■  Reorder  Quantity  and  Reorder  Point 

■  Automatically  runs  multiple  analyses 


Summary  of  Simulator 
Capabilities 

■  Bin  Transfer  Evaluation 

■  Evaluate  best  bin  to  use  for  transfer 


■  Demand  Rate  Profile 

■  Demand  Rate  varies  over  time 

■  Surge  modeling 


I  nputs 

■  Text  file 


■  Simulates  any  number  of  bins  with  any 
varying,  or  the  same,  attributes 

■  Uses  Q,r  values  from  optimization 


I  nputs 


■  Example  data  file 


NSN 

BSL 

BIN 

Q 

R 

LMU 

LSIG 

DMU 

DSIG 

CPI 

DQ 

SINVLOW 

SINVHIGH 

5305003383288 

210 

C069 

88.00 

95.00 

7.60 

22.80 

0.55 

3.28 

3.12 

1.00 

50.00 

150.00 

5305003383288 

150 

A071 

159.00 

222.00 

7.60 

22.80 

0.15 

0.88 

3.12 

1.00 

100.00 

300.00 

■  This  will  automatically  simulate  two  bins 


Models 


■  Read  input  and  invoke  bin  agent  to 
simulate  operation 


■  Simulate  bin  operation  for  each  bin  defined 

■  Stage  1 1  replicated  automatically  for  each 
bin 


Models  -  Read  Data  and 
agent/ simulation  (Stage 


invoke  bin 

I) 


JO  1,  REAJDIDK  >  0 


Models  -  Initialize  Bin 
Agent(  Stage  1 1 ) 


Models  -  Demand  Agent(  Stage  1 1 ) 


RLOGN(DMU,DSIG) 


Models 


Review  Bin  Agent( Stage  1 1 ) 


TBR 


Models  -  Stat  Collection  Agent 
( Stage  1 1 ) 


•Collect  Stats  on  number  of  orders  per  run  and  clocks  per  run 


TTFIN-TNOW-O.Ol  ^ - 

- ►-{SlUMORDERS 


"OrdPerRun" 


-►-("numcldcks  ■ 


"ClocksPerRun" 


COLSTATS 


^{^INF ^)-A/W» 


Outputs 

■  Statistical  collection  for  all  Bin  based 
performance  measures 

■  Includes: 


■  Clock  Time 

■  Orders  per  Run 

■  Clocks  per  Run 

■  Inventory 

■  I  nventory  Cost 

■  Service  Level 


Outputs  -  Example  Report 

(Produced  for  each  Bin) 


**  AweSim!  MULTIPLE  RUN  SUMMARY  REPORT  ** 
Subnetwork  :  BINSIM 
Instance  :  175_Y676 


**  OBSERVED  STATISTICS  for  BINSIM  175_Y676  ** 


Label 

Mean 

Standard 

Standard 

Minimum 

Maximum 

Value 

Deviation 

Error 

Average 

Average 

Value 

Value 

CLOCK  TIME 

3.380 

5.844 

0.826 

0 . 000 

21.402 

Starting  Inv 

386.020 

240.490 

34.010 

3 . 000 

691 . 000 

OrdPerRun 

15.200 

0.670 

0.095 

14 . 000 

17 . 000 

ClocksPerRun 

0.400 

0.571 

0.081 

0 . 000 

2 . 000 

**  TIME- 

-PERSISTENT 

STATISTICS  for 

BINSIM  175_ 

_Y67 6  ** 

Label 

Mean 

Standard 

Standard 

Minimum 

Maximum 

Value 

Deviation 

Error 

Value 

Value 

Bin  Inventory 

735 .897 

39.500 

5.586 

622 .568 

805 . 929 

Service  Rate 

0 . 990 

0 .017 

0.002 

0 . 941 

1 . 000 

Inventory  Value 

1103 . 846 

59.250 

8.379 

933.852 

1208 .894 

Virtual  Bin  Consolidation 


■  Combine  demand  for  groups  of  bins 
with  the  same  NSN  and  calculate  Q,r  for 
group  using  optimization  spreadsheet. 

■  Combines  and  reduces  variance  of 
multiple  bin  processes 

■  Calculate  new  demand  parameters 
based  on  number  of  bins  and  bin  details 

■  Simulated  over  50  years  of  operation 

im   m 


Bin  Consolidation  -  Case  I 


Two  Bins  -  Identical 


■  Q  =  281 
-  R  =  303 

Combined 

■  Q  =  490 

■  R  =  557 


Bin  Consolidation  -  Case  I 
Data  Files 


■  Separate 


NSN 

BSL 

BIN 

Q 

R 

LMU 

LSIG 

DMU 

DSIG 

CPI 

DQ 

SINVLOW 

SINVHIGH 

5305003383288 

175 

Y676 

281.00 

303.00 

7.67 

10.58 

0.10 

0.20 

1.50 

1.00 

0.00 

427.00 

5305003383288 

176 

Y677 

281.00 

303.00 

7.67 

10.58 

0.10 

0.20 

1.50 

1.00 

0.00 

427.00 

■  Combined 


NSN 

BSL 

BIN 

Q 

R 

LMU 

LSIG 

DMU 

DSIG 

CPI 

DQ 

SINVLOW 

SINVHIGH 

5305003383288 

175 

Y676 

490.00 

557.00 

7.67 

10.58 

0.05 

0.14 

1.50 

1.00 

0.00 

700.00 

Results  -  Case  I 


1  Large  Bin 

2BINA 

2BINB 

2  Separate  Bins 

Start  Inv 

374.78 

214.00 

193.80 

407.80 

Inv 

530.90 

343.46 

337.40 

680.86 

InvVal 

796.35 

515.19 

506.11 

1021.30 

Clock  Time 

6.03 

3.20 

3.29 

6.49 

OrdPerRun 

19.22 

13.54 

13.52 

27.06 

CLocksPerRun 

0.96 

0.54 

0.46 

1.00 

Service  Rate 

0.98 

0.99 

0.97 

0.98 

M±//rlf%«r 


Results  -  Case  I 


Results  -  Case  I 


30.00 


25.00 


20.00 


15.00 


10.00 


5.00 


0.00 


□  1  Large  Bin 
■  2  Separate  Bins 


Clock  Time 


OrdPerRun 


Performance  Measures 


Results  -  Cas 


1.00  | 

0.90 

0.80 

0.70 

0.60 

0.50 

0.40 

0.30 

0.20 

0.10 

0.00  - 

CLocksPerRun 


Performance  Measures 


I 


□  1  Large  Bin 

□  2  Separate  Bins 

Service  Rate 


Results  -  Case  I 


■  Ordering  over  two  bins  reduces  number 
of  orders,  inventory  and  costs,  while 
increasing  performance 


■  Even  simple  procedure  is  effective 
(managing  group  of  bins  as  single  bin) 


Bin  Consolidation  -  Case  II 


Four  Bins  -  Identical 


■  Q  =  281 

-  R  =  303 

■  Combined 

-  Q  =  543 

■  R  =  1086 


Bin  Consolidation  -  Case  1 1  Data 
Files 


■  Separate: 


NSN 

BSL 

BIN 

Q 

R 

LMU 

LSIG 

DMU 

DSIG 

CPI 

DQ 

SINVLOW 

SINVHIGH 

5305003383288 

175 

Y676 

281.00 

303.00 

7.67 

10.58 

0.10 

0.20 

1.50 

1.00 

0.00 

427.00 

5305003383288 

176 

Y677 

281.00 

303.00 

7.67 

10.58 

0.10 

0.20 

1.50 

1.00 

0.00 

427.00 

5305003383288 

177 

Y678 

281.00 

303.00 

7.67 

10.58 

0.10 

0.20 

1.50 

1.00 

0.00 

427.00 

5305003383288 

178 

Y679 

281.00 

303.00 

7.67 

10.58 

0.10 

0.20 

1.50 

1.00 

0.00 

427.00 

■  Combined 


NSN 

BSL 

BIN 

Q 

R 

LMU 

LSIG 

DMU 

DSIG 

CPI 

DQ 

SINVLOW 

SINVHIGH 

5305003383288 

175 

Y676 

543.00 

1086.00 

7.67 

10.58 

0.03 

0.04  [ 

1.50 

1.00 

0.00 

1500.00 

Results  -  Case  1 1 


1  Large  Bin 

4BINA 

4BINB 

4BINC 

4BIND 

4  Sep  Bins 

Start  Inv 

719.00 

200.000 

231.000 

221.000 

217.000 

869.000 

Inv 

898.00 

340.000 

344.000 

336.000 

341.000 

1361.000 

InvVal 

1348.00 

511.000 

516.000 

504.000 

512.000 

2043.000 

Clock  Time 

6.00 

4.600 

2.600 

2.700 

2.700 

12.600 

OrdPerRun 

27.00 

13.500 

13.600 

13.400 

13.600 

54.100 

CLocksPerRun 

1.40 

0.560 

0.058 

0.520 

0.660 

1.798 

Service  Rate 

0.968 

0.986 

0.991 

0.971 

0.989 

0.984 

M±//rlf%«r 


Results  -  Case  1 1 


Start  Inv 


InvVal 


□  1  Large  Bin 

□  4  Sep  Bins 


Performance  Measures 


Results  -  Case  1 1 


60.00 


50.00 


40.00 


30.00 


20.00 


10.00 


0.00 


□  1  Large  Bin 
■  4  Sep  Bins 


Clock  Time 


OrdPerRun 


Bin  Consolidation  -  Case  I  I- 
Results 


■  Ordering  over  4  bins  is  even  more  effective 
in  reducing  variance  of  bin  management 
process  and  reducing  inventory/ increasing 
service  rate 


Conclusions 


■  Simulation  has  proved  to  be  an 
effective  tool  in  managing  large  scale 
inventory  systems 

■  Good  Forecasting  tool 

■  Good  analysis  tool 

■  Can  be  made  transparent  to  non-technical 
users 
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National  Science  Foundation  in  Washington,  DC,  where  he  directed 
programs  in  Production  Systems,  Engineering  Design,  and  Operations 
Research.  Dr.  Grant  was  instrumental  in  the  startup  and  development  of 
two  I  ndustrial  Engineering  software  companies:  Pritsker  Corporation  and 
FACTROL.  Dr.  Grant  is  a  Fellow  of  the  I  nstitute  of  I  ndustrial  Engineers  and 
is  a  member  of  the  following  societies:  I NFORMS,  Tau  Beta  Pi  and  The 
Institute  of  American  Entrepreneurs.  Dr.  Grant  received  his  Ph.D.  from 
Purdue  University  in  I  ndustrial  Engineering  in  1980. 


Eugene  Beardslee 


Eugene  Beardslee  is  the  software  system  architect  of  the  I  ndustrial  Prime 
Vendor  (I  PV)  project's  enterprise  system:  SCOPTI MA  Supply  Chain  Software. 
For  the  past  five  years,  Mr.  Beardslee  has  managed  and  directed  the  design, 
development,  implementation  and  maintenance  of  all  aspects  of  this 
logistics  support  software.  Recipient  of  the  2003  SAI C  Achievement  Award 
for  Excellence  in  Program  Performance:  Technology  Development  and 
Analysis,  he  has  guided  three  successful  Manufacturing  Production  and 
Engineering  research  projects  focused  on  improved  inventory  and 
replenishment  process  performance.  Mr.  Beardslee  has  over  18  years 
experience  in  the  production  of  software  systems  with  over  12  years 
designing  Oracle  software  systems.  He  is  a  member  of  the  I EEE  Computer 
Society,  the  Association  for  Computing  Machinery  and  the  I  nstitute  of 
Industrial  Engineers.  Mr.  Beardslee  received  his  Bachelor  of  Science  degree 
in  Computer  Science  in  1988,  and  his  Master  of  Arts  in  Computer  Resources 
nd  I  nformation  Management  in  1997. 
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Bruce  A.  Boyd 
Associate  Technical  Fellow 
The  Boeing  Company 


Copyright  2005  The  Boeing  Company.  26  October  2005  NDIA  Systems  Engineering  Conference 


Problem  Statement 


Many  plans  for  system  development  projects  do  not  reflect  an 
understanding  of  the  development  life-cycle  strategies  for  the 
system  elements  being  defined,  developed  and  integrated. 


This  frequently  leads  to: 

•  Simple,  success-oriented  (i.e.,  fantasy)  plans 

•  Frequent  project  rebaselining  and  replanning 

•  Delivering  less  than  desired  capability 

•  Adding  additional  development  cycles 

•  Work-arounds  for  late  subsystems 
and  components 

•  Re-architecting  the  system 
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Proposed  Solution 


Identify  development  life-cycle  strategies  for  the  overall 
system  and  for  each  major  subsystem  and  component  being 
developed. 


Integrate  these  life-cycle  strategies  into  the  plans  for 
developing  each  subsystem  and  component  comprising  the 
system. 
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A  Life-Cycle-Based  Planning  Process 


1. 

2 

3, 

4. 

5. 

6, 
7. 


8. 


the  overall  system  architecture 

Identify  the  applicable  life-cycle  strategies 

Define  the  appropriate  life-cycle  phases 

Define  the  iterations  for  each  development 

Identify  the  allocation  and  integration  events 

Identify  the  processes  and  work  products 

Define  the  overall  project  plan  and  schedule 
(e.g.,  IMP  and  IMS) 

Define  tasks,  estimates,  staffing,  etc. 
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1.  Define  the  System  Architecture 


<Aj 


G 


Based  upon  an  initial  review  and  analysis  of  the  customer’s 
desired  capabilities,  define  an  overall  system  architecture  (design 
concept). 


This  architecture  should  define  the  levels  of  system  elements  to 
be  developed... 


System  of  Systems 

Systems 

Segments 

Subsystems 

Software  Items 

Assemblies 

Components 

Parts 
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System  Development  Layers 


The  following  examples  will  use  this  simple  3-layer  architecture: 

System  Layer 


System 

Requirements 

Development 

System 

Design 

Subsystem 

Development 

System 

Integration 

System  Test 

Allocated  Requirements 

a 

Tested  Subsystems 

Subsystem  Layer 

■ 

— ► 

Subsystem 

Requirements 

Development 

Subsystem 

Component 

Subsystem 

Subsystem 

Design 

Development 

Integration 

Test 

- 

Allocated  Requirements 

a 

Tested  Components 

Component  Layer 

d^  ■ 

— ► 

Component 

Requirements 

Development 

Component 

Component 

Component 

Component 

Design 

Implementation 

Integration 

Test 

- 
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Work  Breakdown  Structure 


0^ 


G 


The  project’s  work  breakdown  structure  (WBS)  should  be 
developed  in  conjunction  with  the  life-cycle  planning  activities. 

*  It  is  assumed  that  the  customer  has  provided  a  high-level 
WBS  (e.g.,  Contract  WBS  or  CWBS)  that  defines  the  high-level 
deliverables  tied  to  contract  line  items  (CLINs). 

•  It  is  the  project’s  responsibility  to  add  needed  detail  to  this 
WBS  to  fully  develop  the  Program  or  Project  WBS  (PWBS) 
that  will  be  the  basis  for  detailed  project  planning  and 
tracking. 


Life-cycle-based  planning  provides  a  mechanism 
for  evolving  the  WBS  to  the  appropriate 
level  of  detail  in  a  systematic  manner. 
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2.  Identify  Life-Cycle  Strategy 


At  each  level  of  system  development  identify  the  life-cycle 
strategy  to  be  followed: 

•  Once-through  (Waterfall) 

•  Incremental 

•  Evolutionary  (Spiral) 

The  project  may  employ  multiple  strategies  for  different  system 
developments,  depending  upon  the  complexities,  risks,  and 
delivery  requirements  for  each  system  element. 


Higher-Level 

Design 


Requirements 
Allocated  to  the 
Element 


Element 

Requirements 

Development 

Element 

Design 

Element 

Implementation 

Higher-Level 

Design 


Requirements 
Allocated  to  the 
Element 


Elc 


Element 

Requirements 

Development 


Build  1 
Design 

Build  1 

Implementation 

Build  1 
Integration 

Build 

Test 

Build  2 
Design 


Build  2 

Implementation 


Build  2 
Integration 


Build  3 
Design 

Build  3 

Implementation 

Build  3 
Integratic 

Higher-Level 

Design 


Higher-Level 

Integration 


Requirements 
Allocated  to  the 
Element 


Tested 
Element  Builds 
Ready  for 
Integration 


Build  1 

Requirements 

Development 

Build  1 
Design 

Build  1 

Implementation 

Build  1 
Integration 

Build  1 
Test 

Development 


Build  3 

Requirements 

Development 


Build  3 
Design 


Build  3 

Implementation 


Build  3 
Integration 


Build  3 
Test 


Requirements  are  Developed  for  each  Build 


Requirements  are  Allocated  to  each  Build 
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Once-Through  Life-Cycle  Strategy 


Higher-Level 

Higher-Level 

Design 

Integration 

Requirements 
Allocated  to  the 
Element 


Tested 
Element 
Ready  for 
Integration 


Element 

Requirements 

Development 

Element 

Element 

Element 

Element 

Design 

Implementation 

Integration 

Test 
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Incremental  Life-CyclPi Strategy 


Higher-Level 

Higher-Level 

Design 

Integration 

Requirements 
Allocated  to  the 
Element 


Tested 
Element  Builds 
Ready  for 
Integration 


Build  1 

Build  1 

Build  1 

Build  1 

Design 

Implementation 

Integration 

Test 

Element 

Requirements 

Development 


Build  2 

Build  2 

Build  2 

Build  2 

Design 

Implementation 

Integration 

Test 

Build  3 

Build  3 

Build  3 

Build  3 

Design 

Implementation 

Integration 

Test 

Requirements  are  Allocated  to  each  Build 
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Evolutionary  Life-Cycle  Strategy 


Higher-Level 

Higher-Level 

Design 

Integration 

Requirements 
Allocated  to  the 
Element 


Tested 
Element  Builds 
Ready  for 
Integration 


Build  1 

Requirements 

Development 

Build  1 
Design 

Build  1 

Implementation 

Build  1 
Integration 

Build  1 
Test 

Build  2 

Requirements 

Development 

Build  2 

Build  2 

Build  2 

Build  2 

Design 

Implementation 

Integration 

Test 

Build  3 

Requirements 

Development 

Build  3 
Design 

Build  3 

Implementation 

Build  3 
Integration 

Build  3 
Test 

Requirements  are  Developed  for  each  Build 
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Example  -  Overall  System  Strategy 


For  the  overall  system,  the  customer  wants  Initial 
Operational  Capability  (IOC)  in  two  years,  and  Final 
Operational  Capability  (FOC)  in  five  years. 

IOC  has  some  minimum  requirements,  but  they 
would  like  as  much  capability  as  can  be  achieved 
with  medium  risk. 

FOC  also  has  some  minimum  requirements,  but  the 
customer  would  like  us  to  attempt  to  include  some 
innovative  capabilities. 


These  needs  dictate  an  Evolutionary  strategy 
at  the  system  level  with  at  least  two  iterations. 
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Example  -  Subsystem  Strategies 


0-' 


G 


•  One  of  the  major  subsystems  is  an  upgrade  to  an  existing 
legacy  subsystem. 

•  The  requirements  are  well  defined  and  it  is  needed  at  full 
capability  for  IOC.  It  will  not  change  for  FOC. 

The  Once-Through  (Waterfall)  strategy 
is  appropriate  for  this  subsystem. 

•  Another  major  subsystem  will  employ  an  innovative 
networking  technology  that  will  require  prototyping. 

•  The  performance  limits  of  this  technology  are  not  yet  known. 

•  IOC  requires  this  subsystem  to  be  functional,  but  FOC 
requires  a  high  performance  level. 


This  subsystem  will  follow  an  Evolutionary  strategy. 
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Example  Bund  Plan 


Overall  System _ ^/\  IOC 

IOC  System  Build 

FOC  System  Build 


Subsystem  1 

IOC/FOC  Build 


Subsystem  2 

Prototype:  Build  0 

IOC:  Build  1 


Build  2 


FOC:  Build  3 
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3.  Derine  the  Life-Cycle? Phases 


0-" 


For  each  development,  identify  the  life-cycle  phases  to  be 
performed  in  each  iteration. 

If  an  element  includes  hardware  and  software  development, 
identify  the  phases  and  iterations  needed  for  each. 

Identify  procurement  and  supplier  management  phases,  as 
appropriate. 


Typical  development  life-cycle 
phases  include: 

■  Requirements  Development 

•  Preliminary  Design 

■  Detailed  Design 

■  Implementation 

•  Integration 

■  Testing  (V&V) 
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About  Life-Cycle* Phases 


Phases  may  be  defined  at  any  or  all  system  levels: 
•  System  of  Systems,  System,  Subsystem,  etc. 


The  following  are  activities  to  be  performed  within  phases,  but  are 
probably  not  themselves  phases: 

•  Requirements  Analysis 

•  Requirements  Validation 

•  Requirements  Management 

•  Verification 

•  Validation 

•  Qualification  Testing 

•  Rework 

•  Product  Release 

•  Project  Planning  (?) 


Copyright  2005  The  Boeing  Company.  26  October  2005  NDIA  Systems  Engineering  Conference 


Example  System  Level?  Phases 


At  the  overall  system  level,  the  following  phases  may  be  defined 
for  the  IOC  Build: 

•  System  Requirements  Development 

•  Preliminary  System  Design 

•  System  Integration 

•  System  Test 

Subphases  may  be  defined  within  phases  to  provide  more 
definition;  e.g., 

•  System  Test  may  be  divided  into... 

•  Qualification  Test 

■  Flight  Test 

■  Operational  Test  (OT&E) 
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Exa m pie  Su b system  Phases 


At  the  subsystem  level,  Subsystem  2  may  define  the  following 
phases  for  the  IOC  Build  1  iteration: 

•  Subsystem  Requirements  Development 
Software  Requirements  Development 
Software  Design 
Software  Implementation 
Software  Integration 
Software  Test 

Hardware  Requirements  Development 
Hardware  Design 
Hardware  Implementation 
Hardware  Assembly  and  Checkout 
Hardware-Software  Integration 
Subsystem  Test 
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Example  Phase  Relationships 


0^ 


Overall  System  -  IOC  System  Build 


System 

Requirements 

Development 


System  Design 


Subsystem 

Development 


IOC 


System  Test 


System 

Integration 


Subsystem  2  -  IOC  Build  1 


Subsystem 

Requirements 

Development 


Software 

Requirements 

Development 

Software 

Design 

Software 

Implementation 

Software 

Integration 

Software 

Test 

Hardware 

Requirements 

Development 

Hardware 

Design 

Hardware 

Implementation 

Hardware  Assembly  & 
Checkout 

Hardware  - 
Software 
Integration 


Subsystem 

Test 
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4.  identify  Iterations 


If  not  done  already,  define  the  iterations  required  for  each 
incremental  and  evolutionary  strategy. 


Considerations: 

•  Complexity 

•  Risk 

•  Maturity 

•  Delivery  Needs 

•  Dependencies 

•  Constraints 


Each  iteration  should  have  a  well-defined  goal. 
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Example  Iteration  Goal 


•  Subsystem  2  FOC  hardware  will  not  be  ready  until  1  year 
before  the  FOC  milestone. 

•  To  reduce  risk,  the  customer  would  like  to  validate  some 
communication  functions  in  an  operational  environment  using 
IOC  hardware. 

•  Subsystem  2  Build  2  will  accommodate  this  validation  by 
providing  a  software-only  build  1  year  into  FOC  development. 


:  O £ 

Iteration  Goal: 

Validate  the  FOC  Communication 
Software  using  the  IOC  Hardware 
Configuration; 
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Exa m pie  Su b system  Iterations 


Subsystem  2  -  Build  2 


IOC  H/W 


SW  Reqts 

Software 

Software 

SW 

SW 

Devel. 

Design 

Implem. 

Integ. 

Test 

Subsystem  2  -  FOC  Build  3 


Hardware  - 

Software 

Subsystem 

Integration 

Test 

(IOC  H/W) 

Subsystem 


SW  Reqts 

Software 

Software 

SW 

SW 

Devel. 

Design 

Implem. 

Integ. 

Test 

Development 

Hardware 

Hardware 

Requirements 

Source 

Development 

Selection 

Hardware  Supplier 
Management 


Hardware 
Acceptance  Test 


Hardware  - 

Software 

Subsystem 

Integration 

Test 

(FOC  H/W) 
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5.  Identify  Allocation  and  Integration  Events 


Define  the  key  points  within  each  level  of  system 
decomposition  when  requirements  will  be  allocated  to 
lower  tier  elements  and  when  lower-tier  elements  will 
be  integrated  into  subsystems  and  systems. 


These  are  good  points 
at  which  to  schedule 
milestone  reviews 
such  as: 

Subsystem  Requirements  Reviews  (SRRs)  and 
Test  Readiness  Reviews  (TRRs). 
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Exa mple>  M ilestone  Events 


Overall  System  -  IOC  System  Build 


System  SFjjR 
Requirements 
Development 


System  Design 


Subsystem 

Development 


IOC 


TRR 


System  Test 


System 

Integration 


Subsystem  2  -  IOC  Build  1 
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Softwar  QRR  .. 
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Requiremt..—  ..  „ 

Development  esign 


Subsyst  SRR 

Requirem _ 

Development  / 


Software 

1 

SoftWc  TRR  [ware 

Implementation 

-“A" 
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Requirements 
Development 


Hardware 

Hardware 

Hardware  Assembly  & 

Design 

Implementation 
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Allocation  and  Integration  Events 


When  scheduling  the  project,  these  events  are  the  key 
points  for  defining  dependencies. 

•  Subsystem  development  depends  upon  allocation 
of  system-level  requirements 

•  System-level  testing 
depends  upon 
completing  the 
integration  of 
subsystems 
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6.  Identify  Processes  anal  Work  Products 


For  each  phase  identified  within  each  system  element’s 
life  cycle: 

•  Identify  the  processes  that  will  be  used  to  perform 
the  activities  in  that  phase 

•  Identify  the  major  work  products  to  be  produced  or 
updated  during  that  phase. 
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Example  Phase  Processes  and  Work  Products 


For  Subsystem  2: 


Phase 

Processes 

Work  Products 

Subsystem 

Requirements 

Development 

•  Standard  Project 
Planning  Process 

•  Standard 
Requirements 
Development 
Process 

•  Standard 
Requirements 
Management 
Process 

*  Project  Plan 

*  Requirements 
Specification 

*  Technical 
Performance 
Measures 

*  Functional 
Architecture 

*  Operational 
Concepts  and 
Scenarios 

*  Requirements 
Traceability 
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7.  Define  the  Project  Plans  and  Schedule 


0^ 


Once  the  project’s  life-cycle  phases  have  been  defined 
in  terms  of  processes  and  work  products,  define  the 
overall  project  plans  and  schedules. 

*  A  typical  way  to  represent  the  overall  planning  with 
dependencies  is  with  an  Integrated  Master  Plan 
(IMP)  and  Integrated  Master  Schedule  (IMS). 

•  However  the  overall  plan  is  documented,  it  should 
be  consistent  with  and/or  traceable  to  the  project’s 
WBS. 
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8.  Develop  Tasks,  Estimates,  Staffing 


© 


Using  the  overall  project  plans  and  schedules,  define 
tasks  that  can  be  used  as  the  basis  for  detailed 
estimating  and  staffing. 

•  The  tasks  can  be  defined  in  terms  of  work 
packages  which  will  be  the  basis  for  earned-value 
management. 

•  Again,  the  tasks  and 
work  packages  should 
be  consistent  with 
the  project’s  WBS. 
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Integration  Considerations 


The  steps  described  here  may  be  performed 
iteratively  or  recursively,  until  the  entire  project 
scope  has  been  defined. 

Life-cycle  planning  should  be  tightly  integrated  with 
project  planning,  requirements  development,  and 
process  set  definition. 


Multiple  levels  of  life-cycle  descriptions  may  be 
developed  all  at  once  or  each  level  may  be 
developed  as  its  corresponding  system  element  is 
defined. 
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Development  life-cycles  should  be  defined  at  all 
levels  of  the  system  architecture 

Life-cycle  strategies,  phases,  and  iterations  should 
be  defined  as  part  of  project  planning 

Engineering  processes  should  be  mapped  to  all 
phases  of  system  development 

Allocation  and  integration  events  define  milestones 
and  dependencies  in  the  project’s  plans 


The  Bottom  Line: 

Use  of  a  life-cycle  planning  process  should 
result  in  project  plans  that  are  more  accurate , 
comprehensive ,  and  lower  risk! 
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Process  for  Evaluating  Logistics 
Readiness  Levels  (LRLs)  for 
Acquisition  Systems 

October  26,  2005 

Engineering  ) 

LogisticsJCost  j 

Aging  Aircraft  Integrated  Product  Team  (AAIPT) 
Elizabeth  Broadus 
Booz  Allen  Hamilton,  Inc. 


Technology 
&  Process 


Agenda 


•  Background  of  Logistics  Readiness  Level  (LRL)  concept 

•  LRL  Defined 

•  LRL  Excerpts 

•  Benefits 

•  Next  Steps 

•  Questions 

•  NAVAI R  Aging  Aircraft  Points  of  Contact 


Background 


•  LRL  concept  initially  based  on  the  DOD  5000.2 
mandated  Technology  Readiness  Levels  (TRL) 
assessment  process 

•  TRLs  provide: 

-  Evaluation  of  critical  technology  maturity 

-  Maturation  plan  (as  needed) 

-  Best  practices/guidelines  for  each  Milestone 

•  MS  B  target  is  TRL  =  6 

•  MS  C  target  is  TRL  =  7 

•  MS  C  preferred  is  TRL  =  8 


Background:  TRL  Definitions 


System  Validated  on  Representative  A/C  Via  OT  . . . 
System  Validated  on  Representative  A/C  Via  DT  . . . 
System  Demo  ~  Dynamic  OP  Flight  Environ  .... 
Sys/Subsys  Demo  ~  Relevant  Lab  Environ  . . . 

Component/Breadboard  ~  Relevant  Environ . 

Component/Breadboard  ~  Lab  Environ . 


TRL  9 
TRL  8 
TRL  7 
TRL  6 
TRL  5 
TRL  4 


Analytical  /Experimental  Proof-of-Concept 

Technology  Concept . 

Basic  Principles . 


TRL  3 


•  System  Completed 

•  Fit  /  Mission  Qual 

•  System/Subsystem 
Development 

•  Tech  Demo 

•  Tech 

Development 

•Research  to  Prove 
Feasibility 

•  Basic  Tech 
Research 


Background 


•  TRLs  provide  an  understanding  of  the  technical  maturity 
without  consideration  of  the  sustainment  of  those 
technologies 

-  TRLs  were  never  intended  to  consider  logistics 

-  LRLs  is  a  new  concept  with  the  intent  to  consider 
sustainment  issues 

•  Logistics  benchmark  system  was  desirable 

-  —10  logistics  elements  that  are  often  interdependent 
and  parallel  are  required  to  successfully  acquire,  field, 
and  support  new  technology 

-  Aid  in  understanding  what  sustainment  is  required  at 
different  time  phases 


LRL  Definition 


•  LRL  intent: 

-  Provide  a  methodology  for  assessing  Logistic 
Element  Readiness  for  technology 

-  Establish  benchmarks  for  programs  at  different 
phases  in  time 

-  Provide  a  management  tool  to  forecast  logistics 
workload,  manpower  requirements,  identify  gaps, 
etc. 

•  NAVAI R  Aging  Aircraft  convened  a  working  group  of 
engineers,  logisticians,  and  program  managers  to  draft 
an  LRL  concept 

-  LRL  concept  is  work  in  progress!!! 

-  I  nitial  phase  was  to  focus  on  technology  insertion 
for  in-service  (post  MS  C)  aircraft  platforms 

-  LRL  evaluated  for  project  (vice  platform) 


Draft  LRL 


•  LRL's  considered  at  the  project  level  (i.e.  technology 
insertion)  for  an  in  service  aircraft 

•  LRL's  evaluated  for  6  project  phases: 

-  Lab  Test/  R&D 

-  Project  Definition  (Fleet  Need/ metrics/ BC/V Decision  to 
proceed) 

-  Project  Development /I  mplementation  (Finalized 
analysis,  change  recommended,  ECP  development. 
Class  1 1  change  development,  RAMEC,  LECP,  other) 

-  Engineering  Validation 

-  Fleet  Verification 

-  Fleet  Use 

•  Answers  question  of  what  has  to  be  done  at  each  project 
phase  for  logistics 


LRL  Excerpt  —  Design  Interface 


Phase 

Lab 

Test/R&D 

phase 

Project 

Definition) 

Project 

Development 

/Implementat 

ion 

Engineering 

Validation 

Fleet 

Verification 

Fleet  Use 

Design 

Interface 

Review  and 

identify 

significant 

design 

interface 

impacts  of 

project  to 

existing 

system  or 

platform  (ex. 

available 

power, 

weight 

constraints, 

etc.)  Create 

POAM  to 

resolve  any 

design 

interface 

issues. 

Existing 

Reliability  and 
Maintainability 

(RAM)  metrics 
reviewed.  Initial 
improvement 
predictions 
determined. 

Design  interface 
issue  resolution  in 
work. 

For  new  designs, 
Reliability 
Centered 
Maintenance 
(ROM)  and 

Failure  Modes 
and  Effects 
Analysis 
(FMECA) 
completed  to 
identify  failure 
modes,  failure 
frequency,  effect 
on  performance, 
and  criticality. 

For  modifications 
to  existing  design, 
RCM  and  FMECA 
reviewed  for 
impacts.  Design 
interface  issues 
resolved. 

Results  of  RCM 
and  FMECA  used 
to  develop  or 
modify  existing 
condition  based 
and  schedule 
based 

maintenance 
tasks.  Results  of 
RCM  and  FMECA 
also  used  to 
update  the  Critical 
Items  list  as 
applicable. 
Technical  data 
updates  drafted 
and  validated. 

Technical 
updates  (such  as 
Maintenance 
Requirement 

Card  changes) 
verified. 

Technical 

updates 
completed  and 
available. 

LRL  Excerpt  —  Training  and  Facilities 


Phase 

Lab  Test 
R&D 
phase 

Project 

Definit 

ion 

Project 

Development 

/Implementati 

on 

Engineering 

Validati 

on 

Fleet 

Verificati 

on 

Fleet  Use 

Training 

Existing  training 
procedur 
es/curric 
ula  and 
training 
plan 
identified 
and 

reviewed. 

Impacts  to  Training 

identified 

Training 

curricula 

changes 
drafted.  As 
required 
changes  to 

Naval 

Training 

Systems 

Plan 

(NTSP) 

drafted. 

Training  curricula 

changes 
updated  post 
validation/verif 
ication  with 
changes  as 
necessary. 
NTSP 
changes 
finalized. 
Changes 
submitted  for 
approval. 

Trainining 

curricula 

updated. 

NTSP 

updated. 

Facilities 

Current  Facilities  reviewed 
and  impacts 
identified.  When 
applicable,  facilities 
modifications  or  new 
requirements  are 
documented  and 
analysis  completed 
for  (in)  adequacy  of 
existing  facilities, 
trade  studies  for 
optimal  new  facility, 
funding  requested. 

As  needed  with 
funding 
available, 

Facilities 

modification 

s  or  new 

facilities 
projects  in 
work. 

Facilities  project 
completed 
and  approved. 

New  or  modified 

Facilities 

completed  . 

LRL  Excerpt  -  DMSMS 


Project 

Develop 

Phase 

Lab  Test 

Project 

merit 

R&D 

Definitio 

/Implem 

Engineering 

Fleet 

phase 

n 

entation 

Validation 

Verification 

Fleet  Use 

DMSMS 

Existing 

DMSMS 
program 
managem 
ent  plan 

reviewed. 

Determine 

the 

technical 

refresh 

strategy 

(2  yr,  4  yr, 
spiral,  etc.) 

New  technology 
evaluated 
to 

determine 

criticality  as 

it  relates  to 

DMSMS. 

Assess 

component 

s  against 

the  tech 

refresh 

strategy. 

Impacts  to 

DMSMS 

plan  or 

metrics 

identified. 

DMSMS 

forecasting 

completed 

for  new 

technology. 

Updates  to 

DMSMS 

management 

plan  drafted. 

Technical 

data 

package 

requirements 

drafted. 

DMSMS 

management 
plan  updated. 
Technical  data 
package  that 
supports 
DMSMS 
mitigation 
strategy 
available. 

Metrics  and 

usage 
monitor 
ed  as 
require 
d. 

Draft  LRL 


•  Evaluated  percentage  of  total  effort  required  at  each 
phase 

•  Graphed  percentage  effort  as  a  function  of  phase  of  the 
project 

•  Percent  effort  is  subjective  number  based  on  efforts 
outlined  in  the  LRL  for  each  element  at  each  phase 

•  Many  other  ways  to  depict  data 
-  3  examples  follow 


Percent  Effort 


Percent  Effort  by  Project  Phase 


100 


Lab  Test/R&D  Phase  Project  Definition 


Project  Development  Engineering  Validation 

Project  Phase 


Fleet  Verification 


Fleet  Use 


60 

50 

40 

30 

20 

10 

0 


Percent  Effort  During  Each  Phase 


□  Technical  Data 

□  Maintenance  Planning 

■  Training 

□  Facilities 

□  Supply  Support 

■  Manpower 
□ PHS&T 

□  Support  Equipment 

□  Computer  Resources 


Lab  Test/R&D  Phase  Project  Definition  Project  Development  Engineering  Fleet  Verification  Fleet  Use 

Validation 


Phase  of  Project 


Percent  Effort 


Percent  Effort  vs.  Project  Phase 
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□  Fleet  Use 

■  Fleet  Verification 

□  Engineering  Validation 

□  Project  Development 

■  Project  Definition 

□  Lab  Test/Ft&D  Phase 
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Logistic  Element 


LRL  Benefits 


•  Benefits  include: 

-  LRLs  will  be  a  template/benchmark  to  measure  readiness 
by  logistic  element  on  a  project  level  basis 

-  Template  can  be  utilized  to  train/mentor  new  logistics 
personnel  (and  engineering  personnel)  in  sustainment 
requirements  for  tech  insertion  projects 

-  LRLs  will  aid  in  planning  manpower/funding/schedule 
requirements  for  projects  as  they  mature  from  project 
concept  to  implementation 

-  LRLs  will  dovetail  with  logistics  risk  assessments  for 
another  perspective 


Next  Steps 


•  Continue  to  collect  input  on  Draft  LRL  concept 

•  Brief  Draft  LRL  concept  to  solicit  further  input 

•  Update/ change  draft  as  needed 

•  Establish  a  working  group  to  expand  the  scope  to 
encompass  aircraft  in  the  entire  lifecycle  vice  limiting  to  in- 
service  aircraft 

•  Apply  and  test  the  process  at  NAVAI R  AAI PT 


Questions? 


AAIPT  Points  of  Contact 


•  AAI PT  Lead,  Bob  Ernst,  301-342-2203, 
robert .  ernst@navy .  mi  I 

•  AAI  PT  Assistant  Program  Manager  for  Logistics,  Harry 
Proffitt,  301-757-0868,  rrelvin.proffitt@navy.mil 

•  AAI  PT  Air  Vehicle  I  PT  lead,  Don  Sheehan,  301-342-0131, 
donald.sheehan@navy.  mil 

•  AAI  PT  Consultant,  Elizabeth  Broadus,  301-862-7049, 
broadus_elizabeth@bah.com 


Backup  Slides 


AAIPT  Organization 

Why  is  an  AAIPT  needed? 
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AAI  FT  Vision: 

-  I  dentify  Problems  -  Quantify  Risk 

-  Provide  Information  to  Program  teams 

-  Advocate  for  Enabling  Technologies 

-  Provide  Standard  Risk  &  Cost  Evaluation 

Tools 

-  Focus  Attention  to  Aging  Aircraft  problems 

-  Leveraged  Funding  to  reduce  cost 


AAIPT  Organization 


AIR  1.0 

AIR  3.0  /  6.0 

AIR  4.0  /  5.0 

AAIPT  LEAD 


STAFF 

FUNCTIONS 


FASTRACK 
SUSTAINMENT 
(S.E.T .) 


>►  Process 


•  Command  Support  Throughout 
NAVAIR 

•  Technical  Expertise 

•  Adequately  staffed  to  meet 
emergent  needs  (Triage) 


LOGISTICS 


COST 


►  Products 


H53 

H-1 

AV-8 

H- 

46 

FINANCES 


CONTRACTS 


JCAA 


Fleet  Focus  Team 


•  Fleet  Focus  Teams  (FFTs)  identify/  communicate,  and  leverage 
engineering  solutions  across  Type  Model  Series  and/or  Service 
Boundaries 


Simply  Stated: 


NAVAL  SEA  SYSTEMS  COMMAND 


Ordnance  Safety  &  Security  Activity 


r. 


Lessons  Learned  with  the  Application  o 

MIL-STD-882D  at  the 


Weapon  System  Explosives  Safety  Review 

Board 


Mary  Ellen  Caro 

Naval  Ordnance  Safety  and  Security  Activity 
Systems  Safety/Joint  Programs 


Presented  to: 

8th  Systems  Engineering  Conference 
26  October  2005 


Providing  Ordnance  Safety  for  our  Warfighters 
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Agenda 


•  WSESRB  Background 

•  MIL-STD-882D  Evolution 

•  MIL-STD-882D  Implications  for 

-  System  Acquisition 

-  System  Safety  Program  Planning 

-  Safety  Program  Execution 

-  Safety  Risk  Management 

•  Conclusion 


la 


NAVAL  SEA  SYSTEMS  COMMAND 


Ordnance  Safety  &  Security  Activity 


WSESRB  Background 


The  WSESRB  was  established  in 
1967  as  a  result  of  several  mishaps 
aboard  aircraft  carriers 


The  purpose  of  the  WSESRB 
is  to  provide  an  independent 
and  technical  review  of  the 
adequacy  of  the  Program’s 
system  safety  program  and 
artifacts 


USS  Oriskany 
(1966) 


USS  Forrestal 
(1967J 
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NAVAL  SEA  SYSTEMS  COMMAND 


Ordnance  Safety  &  Security  Activity 


WSESRB  Authority 


DODI  5000.2  Para  E7. 7 

-  PM  shall  identify,  evaluate  and  manage  safety  and  health 
hazards 

-  Explains  the  process  for  accepting  risk 

SECNA  VINST  5000.2C 

-CNO  may  establish  system  safety  advisory  boards  (7.3.3) 

-  WSESRB  is  primary  explosives  safety  review  prior  to 
DT/OT  and  Milestones  (5.2.1 .4.2) 

SECNA  VINST  5100. 10H 

-  Directs  CNO/CMC  to  establish  safety  programs 

OPNA  VINST 8020. 14/MCO  P8Q20. 1 1 

-  Explosives  Safety  Policy 

-  Tasks  COMNAVSEASYSCOM  to  establish  WSESRB 

NA  VSEAINST 8020.6D 

-  Defines  WSESRB  process  and  procedures  4 


Who  is  the  “WSESRB” 


NAVAL  SEA  SYSTEMS  COMMAND 


Ordnance 


safety  &  Security  Actmty  |  Explosives  Safety  Program  Policy  Flow  and  Membership 


OPNAVINST  8020.14/ 
MCO  P8020.ll 


Chair 

& 

Secretariat 


Naval  Safety 
Center 

Member 


Navy  Environ 
Health  Center 

Member 


Fleet 

Members 


Recently  Established 
A  Flag  Level 
Review  Process 


Naval  Sea 
Systems  Command 

PT 

NOSSA 

Weapon  System 
Explosives  Safety 
Review  Board 
(WSESRB) 


NAVSEAINST 
8020.6 


NAVAIR 

Members 


OPNAV 

Member 


NAVSEA 

Members 


EOD 

TECHDIV 

Member 

MARCOR 

SYSCOM 

Member 
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Acquisition  Life-Cycle 


r. 


WSESRB  reviews  occur  throughout  that  life-cycle 


Phases 


Work 

Efforts 


A  A 


A 


Concept 

Technology 

System  Development  5 

Production  & 

Operations  & 

ReflNuniLid 

Dev  o  fop  mem 

Pemcrifciratloh 

Deploy  mm 

Support 

System  System 

.  F Li! Rate 

inip  pi!X|m:lKi|i 

Susta  n  merit 

Integration  Demonstration 

Disposal 

L  Concept 

r 

A  OfiElgn 

V  Readiness 

10  TE  *FRP 

■DeiKion 
"  Review 

IOC 


FOC 


Anliviliftfi  —  Pm  SyKtAmfc  Acquisition  -V 
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Transition  to  MIL-STD-882D 


Ordnance  Safety  &  Security  Activity 


Developed  as  result  of  acquisition  reform 

-  Converted  to  Standard  Practice  document 

Eliminated  system  safety  tasks 

-  “What  to  do”  not  “How  to  do  it” 

Example  Mishap  Risk  Index  and  defined 
High,  Serious,  Medium  and  Low  risks 

-  Agreement  with  DoDI  5000.2 

-  Ability  to  tailor  to  specific  programs 

Requirement  for  Closed  Loop  Hazard 
Tracking 


Providing  Ordnance  Safety  for  our  Warfighters 
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Ordnance  Safety  &  Security  Activity 


MIL-STD-882D  Process 


l 


Eight  basic  steps  to  the  MIL-STD-882D 
Standard  Practice 


-  Documentation  of  the  System  Safety  approach 

-  Identification  of  hazards 

-  Assessment  of  mishap  risk 

-  Identification  of  mitigation  measures 

-  Reduction  of  mishap  risk  to  acceptable  level 

-  Verification  of  mishap  risk  reduction 

-  Acceptance  of  residual  risk 

-  Hazard  tracking 


Providing  Ordnance  Safety  for  our  Warfighters 
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Ordnance  Safety  &  Security  Activity 


System  Acquisitions 


r. 


MIL-STD-882D  calls  for  System  Safety 
Program,  but  eliminated  tasks 

-  No  tasks  to  identify  in  solicitation 

“The  bidder  shall  execute  a  system  safety  program  in 
accordance  with  MIL-STD-882D” 


“System  Safety  Hazard  Analysis  shall  be  provided  xx  days 
prior  to  DRR” 

-  Bidders  propose  safety  programs  for  best 
competitive  advantage 

-  Proposals  may  vary  widely  in  planned  system 
safety  program 

-  Potential  ambiguities  between  buyer  and  seller  in 
program  execution 


NAVAL  SEA  SYSTEMS  COMMAND 


Ordnance  Safety  &  Security  Activity 


System  Acquisitions 


r. 


Lessons  Learned 


-  Solicitation  needs  to  be  as  specific  as  possible 
and  identify  types  of  system  safety  efforts  required 
of  the  developer  (e.g.,  system  safety  program 
plan/POA&M,  hazard  analyses,  hazard  testing, 
certification  requirements) 

-  For  Navy  ordnance  and  weapon  programs,  there 
are  many  required  tests  and  analyses  that  need  to 
be  identified  in  solicitation  documents 

-  Safety  should  to  be  part  of  Source  Selection 
Criteria  and  participate  in  proposal  evaluations 
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System  Safety  Program 

Planning 


•  DoDI  5000.2  only  requires  a  PESHE 

•  MIL-STD-882D  requires  system  safety 
program  planning,  but  no  longer 
identifies  a  task  for  the  System  Safety 
Program  Plan  (SSPP) 

•  Solicitation  may  or  may  not  require  an 
SSPP  to  be  submitted  with  the  proposal 


Ordnance  Safety  &  Security  Activity  | 
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System  Safety  Program 

Planning 


•  Lessons  Learned 

-  A  specific  system  safety  plan  needs  to  be 
developed  for  the  program  including 
identification  of  responsibilities,  schedules, 
safety  analyses,  safety  testing.  Typically 
SSPPs  are  still  being  prepared. 

-  For  large  complex  programs,  the 
Government  should  develop  a  System  Safety 
Management  Plan  to  identify  how  project 
safety  efforts  are  aligned  and  integrated. 
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NAVAL  SEA  SYSTEMS  COMMAND 


Ordnance  Safety  &  Security  Activity 


Safety  Program  Execution 


r. 


Integrated  Product  Process  Development 
structure  applied  almost  universally 


•  Concurrent  engineering  requires  real  time 
safety  participation 
-  Hazard  identification 


-  Hazard  characterization 

-  Prioritization  of  hazards 

-  Identification  of  hazard  mitigation 

-  Implementation  and  verification  of  hazard  risk 
mitigation 

•  Collaborative  effort  with  Design  IPTs 
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Mishap  Relationships 


NAVAL  SEA  SYSTEMS  COMMAND 


Ordnance  Safety  &  Security  Activity 


r 


Death,  Injury,  Illness,  Equipment 
Loss,  Equipment  Damage, 
Environmental  Damage 


The  point  at  which  the  Inadvertent 
Release  of  Energy  Occurred 


A  Condition  that  exists  within  the 
system  that  could  lead  to  a  TLMs 


Hazard  Causal  Factors 


Element  within  the  system 
design,  implementation,  or 
y  operation  that  leads  to  a 
hazard 


Providing  Ordnance  Safety  for  our  Warfighters 
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Ordnance  Safety  &  Security  Activity 


Safety  Program  Execution 


Hazard  Analysis  tasks  of  MIL-STD-882C  have  been 
eliminated  in  MIL-STD-882D.  However,  these  tasks 
lead  the  safety  practitioner  through  a  logical  sequence 
of  hazard  identification/mitigation: 


-  Preliminary  Hazard  List/Analysis  (PHL/PHA)  identifies  top  level 
hazards  for  further  development 

-  Safety  Requirements/Criteria  Analysis  (SR/CA)  identifies  safety 
requirements  that  can  be  mapped  to  their  allocated  subsystems 

-  Subsystem  Hazard  Analysis  (SSHA)  further  evaluates  hazards 
associated  with  identified  subsystems 

-  System  Hazard  Analysis  (SHA)  identifies  hazards  of  interfacing 
subsystems/outside  systems 

-  Operating  and  Support  Hazard  Analysis  (O&SHA)  identifies 
those  hazards  associated  with  operations  and  maintenance 


NAVAL  SEA  SYSTEMS  COMMAND 


Ordnance  Safety  &  Security  Activity 


Safety  Program  Execution 


r. 


Lessons  Learned 


-  Safety  practitioner  needs  to  step  back  from 
day-to-day  IPT  activities  to  ensure  that 
correct  aspects  of  safety  analyses  are 
being  conducted 

-  Safety  practitioner  needs  to  ensure  the 
scope  of  all  the  hazard  analysis  types  has 
been  covered  within  the  program  execution 
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Ordnance  Safety  &  Security  Activity 


Safety  Program  Execution 


r. 


Lessons  Learned 


-  Doing  the  system  safety  work  doesn’t 
necessarily  mean  producing  the  specific 
hazard  analysis  documents 

-  Not  having  to  produce  the  specific  hazard 
analysis  documents  doesn’t  mean  not 
having  to  do  the  system  safety  work 
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Ordnance  Safety  &  Security  Activity 


Safety  Program  Execution 


r. 


Lessons  Learned 


-  Hazard  tracking  systems  are  becoming  more 
important 

•  Many  are  web  based  so  everyone  has  access 

•  Repository  for  all  identified  hazards 

•  Real  time  tool  that  can  capture  work  on-going 
within  IPTs 

•  Data  base  formats  allow  manipulation  of  data  to 
produce  information 

•  Tool  for  development  of  System  Safety  Hazard 
Analysis  deliverable  documents 
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Safety  Program  Executio 


•  Software  safety  process  heavily 
dependent  on  identification  of 
safety-related  requirements  and 
assessment  of  criticality 
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NAVAL  SEA  SYSTEMS  COMMAND 


Software  Criticality  Matrix 


SOFTWARE  CONTROL  CATEGORY 

MISHAP  SEVERITY  POTENTIAL 

Catastrophic 

Critical 

Marginal 

Negligible 

Autonomous 

SHRI1 

SHRI1 

SHRI2 

SHRI4 

Semi-Autonomous 

SHRI1 

SHRI2 

SHRI3 

SHRI4 

Semi-Autonomous  with  Redundant  Back- 
Up 

SHRI2 

SHRI3 

SHRI4 

SHRI4 

Influential 

SHRI3 

SHRI3 

SHRI4 

SHRI4 

No  Safety  Involvement 

No  Safety  Analysis  Required. 

-  Safety  verification  requires  requirements  analysis,  design  analysis,  code  analysis 
and  safety  specific  testing 

Serious  Risk  -  Requires  requirements  analysis,  design  analysis  and  in-depth  safety  specific 
testing 

Medium  Risk  -  Requires  requirements  analysis  and  safety  specific  testing 
Low  Risk  -  Requires  requirements  analysis  and  standard  testing  process 


Providing  Ordnance  Safety  for  our  Warfighters 
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Software  Integrity  Matri 


Phase 

SRI 

DESIGN 

CODE 

UNIT  TEST 

INTEGRATING 
UNIT  TEST 

SYSTEM 

INTEGRATION 

SRI  1 

High  Risk 

•  Design  Team  Review 

•  Safety  Review 

•  SCF  Linked  To  SW  Rqmts 

•  SCF  Linked  to  Design 
Architecture 

•  Fault  Tolerant  Design. 

•  Design  Code  Walkthrough 

•  Independent  Code  Review 

•  Safety  Code  Analysis 

•  SCF  Code  Review 

•  Safety  Fault  Detection, 

Fault  Tolerance 

•  Test  Case  Review 

•  Independent  Test  Review 

•  Failure  Mode  Effect 

Testing 

•  100%  Thread  Testing 

•  Safety  Test  Result  Review 

•  Test  Case  Review 

•  Independent  Test  Review 

•  Failure  Mode  Effect 

Testing 

•  100%  Regression  Testing 

•  Sj*£^  ykt  Result  Review 

•  Test  Case  Review 

•  Independent  Test  Review 

•  Failure  Mode  Effect 

Testing 

•  100%  Regression  Testing 

•  Safety  Test  Result  Review 

SRI  2 

Serious  Risk 

•  Design  Team  Review 

•  Prioritized  Safety  Review 

•  SCF  Linked  To  SW  Rqmts 

•  SCF  Linked  to  Design 
Architecture. 

•  Design  Code  Walkthrough 

•  Safety  Code  Analysis  for 
Prioritized  Modules 

•  SCF  Code  Review 

•  Safety  Fault  Detection, 

Fault  Tolerance 

•  Test  Case  Review 

•  Independent  Test  Review 

•  Failure  Mode  Effect  \  \ 

Testing  \ 

•  100%^h^  ^  j  \ 

•  Sjy  \es\  >eview\ 

SRI  3 

Medium  Risk 

•  Design  Team  Review 

•  Limited  Safety  Review 

•  Safety-Related  Functions 
Linked  to  Design 

•  Design < 

•  Safety! 
Prioritiz 

•  SCF  Co 

•  Safety  f 

Cod 

Cod 

:ed  P 

dc^ 

e  Walkthrough, 
e  Analysis  for\ 
vlodu^N.  ] 

Revi\  \sN 

Dete\ 

nee  \  /A 

SRI  4 

Low  Risk 

•Design  Team  Review/^  ^ 

•  Minimal  Safety  Review  V 

•  Normal  Software  Design. 

Process  IAW  SDP  \ 

1 

SRI  5 

No  Safety 
Risk 

•  Normal  Software  Design 
Activity  IAW  the 

Software  Development 

Plan 

•  Normal  Software  Code 
Activity  IAW  the 

Software  Development 

Plan 

•  Normal  Software  Unit 

Test  Activity  IAW  the 
Software  Development 

Plan 

•  Normal  Software  Unit 
Integration  Test  Activity 

IAW  the  Software 
Development  Plan 

•  Normal  Software  System 
Integration  Test  Activity 

IAW  the  Software 
Development  Plan 
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[naval  sea  systems  command 


Ordnance  Safety  &  Security  Activity 


Safety  Program  Executio 


Proposed  revision  to  MIL-STD- 
882D  introduces  concept  of  relating 
safety  criticality  of  software  to 
safety  integrity  levels  similar  to 
DO  178B 


•  Different  levels  of  rigor  in  the 
design,  review,  analysis  and  test 
efforts  for  varying  levels  of  safety 
criticality 
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Ordnance  Safety  &  Security  Activity  | 


L 


System  Safety  Risk 
Management 


•  MIL-STD-882D  addresses  Mishap 
Risk  vice  MIL-STD-882C  Hazard 
Risk 


•  Higher  level  of  abstraction 
associated  with  residual  risk 

-Many  hazards  that  can  result  in  the 
same  mishap 
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NAVAL  SEA  SYSTEMS  COMMAND 


Ordnance  Safety  &  Security  Activity 


Mishap  Risk  Index 


t 


HAZARD  CATEGORIES 


FREQUENCY 

OF 

OCCURRENCE 

(A)  Frequent 

(B)  Probable 

(C)  Occasional 

(D)  Remote 


i 


ii 


CATASTROPIC  CRITICAL 


III 

MARGINAL 


(E)  Improbable 


7  (3A) 
9  (3B) 


11  (3C) 
1  4  (3D) 

1  7  (3E> 


IV 

NEGLIGIBLE 


1  3  <4A> 
1 6  <4B> 
1 8  <4C> 
1  9  (4D) 

20  <4E> 


SERIOUS  (PEO) 


MEDIUM  (PM) 


LOW  (PM) 
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Providing  Ordnance  Safety  for  our  Warfighters 


System  Safety  Risk 
Management  ^ 

•  Lessons  Learned 

-  Mishap  Risk  Index  needs  to  be  tailored  for 
different  applications,  but  most  programs  default 
to  the  identified  MRI  in  MIL-STD-882D. 

-  With  Residual  Risk  being  captured  at  the  Mishap 
vice  Hazard  level,  strategy  for  dealing  with 
cumulative  risk  associated  with  many  hazards 
should  be  identified. 


Ordnance  Safety  &  Security  Activity  | 


L 


Providing  Ordnance  Safety  for  our  Warfighters 
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NAVAL  SEA  SYSTEMS  COMMAND 


Ordnance  Safety  &  Security  Activity 


Conclusions 


l 


Acquisition  reform  and  MIL-STD-882D  have 
changed  the  way  System  Safety  is  performed 

Requires  more  understanding  and  thought  up 
front  to  ensure  the  system  safety  program  is 
properly  structured 

Requires  vigilance  to  ensure  full  scope  of 
system  safety  effort  is  accomplished  vice  only 
those  issues  identified  in  IPT  meetings 


•  Has  fostered  collaborative  efforts  between 
system  safety,  systems  engineering,  software 
engineering  and  design  engineering  on  many 
programs 


Providing  Ordnance  Safety  for  our  Warfighters 
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Areas  to  be  Covered 


1.  Background 

2.  Design  to  Cost  and  Cost  As  an  Independent 
Variable 

3.  The  DTC  Process 

4.  The  DTC  Metric 

5.  Lessons  Learned 

6.  Going  Forward 


Background 

•  The  Engineering  Effectiveness  Metrics  initiative  grew  out  of 
RMS’s  desire  to  reduce  the  costs  and  cycle  times  necessary  to 
design,  develop  and  build  products  that  work  right  the  first 
time. 

•  To  support  these  goals,  an  Engineering  Effectiveness  Metrics 
Team  developed  three  primary  metrics: 

*  On-Time  Delivery  Performance 

*  First-Presentation  Yield 

*  Design  To  Cost  (DTC) 

•  Today’s  focus  is  on  the  creation  of  the  DTC  Metric,  its  purpose 
and  use  at  RMS. 

•  The  DTC  metric  is  designed  to  allow  business  unit  management 
to  quickly  review  program(s)  progress  and  status  towards 
meeting  their  affordability  commitments. 


Background: 

DTC  &  CAIV  at  Raytheon  Missile  Systems 


•  DTC  and  CAIV  are  blended  into  Business  Development  under  the 
heading  of  Affordability. 

•  Within  the  process  at  RMS: 

•  Defined  cost  targets  are  assigned  to  each  IPT 

•  Focus  is  on  identified  cost  drivers 

•  Cost  vs  performance  tradeoffs  are  conducted  that  lead  to  best 
value  solutions 

•  Metrics  are  determined  and  reported  accordingly 

•  Each  design  choice  is  evaluated  simultaneously  for  both  cost 
and  benefit 

•  CAIV  begins  before  Concept  Exploration  and  remains,  with  DTC, 
vigorous  throughout  product  development 

CAIV  -  Cost  As  an  Independent  Variable 


Raytheon 


DTC 
as  a 

Management  Control 

System 


DTC  -  A  Management  Control  System 


Management  Control 
Systems  are  put  in 
place  to  direct 
targeted  activity 
toward  achievement 
of  the  desired 
results. 


Management  Control  Process 

DTC  Process 

Set  goals  and  performance  measures 

Sets  AUPC  Goal  as  part  of  DTC  Plan 

Measures  achievement 

Prepares  current  cost  estimate 

Compares  achievement  with  goals 

Current  estimate  vs.  DTC  goal 

Computes  the  variances  as  the  result  of  the  preceding  comparison 

Estimates  system  and  subsystem  variances 

Reports  variances 

Reports  $  Data 

Determines  the  cause(s)  of  the  variances 

Cost  Drivers,  spec,  risk,  etc. 

Takes  action  to  eliminate  variances 

Action  plan:  changes 

Follow-up  to  ensure  that  goals  are  met 

Repeats  at  interval  per  plan 

Our  DTC  Process 


Requirements  Flowchart 


Design  to  Cost 

is  a  continuous 
iterative  process 
that  begins  at  the 
top  level  with  a 
product 

requirement  that 
includes  cost  as  a 
major  priority  and 
then  seeks  to 
optimize  the  entire 
product  while 
allocating 
requirements 
down  to  all  levels 


Design  Review  As  Appropriate  to  Phase 


r  Iterative  Cost/Performance  Tradeoffs 


Seven  Steps  to  an  Affordable  Design 


REQUIREMENTS 


■••.jliptfci  it- 


UNDER ST ANn 
CUSTOMER 
COST  AND 
PERFORMANCE 
REQUIREMETS 


Allocate 

Cost 

and  Other 
Requirements 


Create 

Balanced  Design 
Approaches 
for  Sub- Products 


Estimate  Cost 
Yield,  Risk, 
Performance 


Rollup 


1.  Understand 


2.  Allocate 


3.  Create 


4.  Estimate 


5.  Aggregate 


O 

a 

I  to 

3 

H 

O 

n 

a 

tj> 

"O 

3 
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The  engineer  must  use  the  following 
7  steps  to  execute  DTC: 

1.  Understand  requirements 

2.  Analyze  functions 

3.  Identify  physical  alternatives  / 
allocate  requirements  /  plan  task 

4.  Design  synthesis 

5.  Cost  Modeling  -  Estimation  & 
Rollup 

6.  Evaluate  -  Meet  or  changes 
requirements? 

7.  Select/Formalize  Design 


Meets 

■  Requirements^ 


accept  DES-TGN 
APPROACH  and 
FLOW 

REQUIREMENTS 
DOWN  TO  SUB- 
PFlO  DUCTS 


6.  Evaluate 


7A  Accept 
OR 

7B  Improve/ 
Iterate 


Plus,  an  often  overlooked  8th  step  to: 
8.  Document  and  report  progress 
towards  meeting  the  cost  goal. 


The  Design  is  Complete  IF 


The  design  is  complete  when  the  customer/contractor  team  has 
accomplished  the  following: 

•  Performed  detailed  cost,  performance,  supportability,  and  risk 
assessments  that  indicate  that  all  final  requirements  will  be  met 
with  levels  of  cost,  schedule  and  technical  risk  acceptable  to  both 
the  customer  and  the  company. 

•  Allocated  all  requirements  to  non  NDI  items  or  specific  custom 
designed  components. 

•  Completed  the  detailed  design  of  all  custom  components. 

•  Successfully  modeled/prototyped  custom  components  and 
assemblies  that  can  drive  cost,  performance,  or  schedule. 

•  Completed  a  thorough  manufacturing  plan  defining  the  approach 
to  the  fabrication  or  procurement  of  all  components  and  the 
assembly,  integration,  and  test  of  the  product  and  each  significant 
sub-product. 

•  Complied  with  all  customer  and  company  requirements  for  ILS, 
support,  review,  documentation,  verification,  scheduling,  warranty, 
and  the  like. 


DTC  Metric 


Metrics  and  System  Engineering 


An  often  asked  question  deals 
within  what  role  do  metrics  have 
within  the  System  Engineering 
Community. 


INTERNATIONAL  COUMZlON  SYSTEMS 


Systems  Engineering 
Measurement  Primer 


A  Basic  Introduction  to  Systems  Engineering  Measurement 
_ Concepts  and  Use _ 

Version  1 .0 
March  1998 


This  document  was  prepared  by  the  Measurement  Working  Group  (MWG)  of  the  International 
Council  on  Systems  Engineering  (INCOSE).  It  was  approved  as  an  INCOSE  Technical  Paper  by 
the  INCOSE  Technical  Board. 


Metrics 


Raytheon 


The  purpose  of  any  metric  is  to  drive  proper  behavior. 

•  Proper  behavior  is  achieved  by  setting,  striving  for,  and  ultimately  reaching  goals. 
A  DTC  metric  is  therefore  one  that  keeps  cost  and  cost  reduction  in  the  forefront. 

•  The  proper  metric  for  DTC  is  one  that  establishes  a  system  cost  goal  for  the 
design  and  that  requires  attainment  of  estimated  production  costs  at  specified 
points  along  a  program  timeline  starting  pre-SDD  and  going  through  production. 

•  By  establishing  cost  goals  for  a  program  (and  its  subsystems)  that  are  time 
phased,  and  constantly  decreasing,  a  program  is  able  to  measure  its  cost 
reduction  effort  toward  the  ultimate  program  cost  goal. 

•  The  DTC  metric  is  measured  as  cost  variance  to  the  required  time-phased  goals. 
Any  variance  to  a  cost  goal  should  precipitate  IPT  action  to  eliminate  the 
discrepancy. 

•  Variances  are  measured  and  reported  at  design  team  meetings  and  program 
reviews.  Efforts  to  eliminate  cost  variances  (the  proper  behavior)  become  part  of 
the  IPT  design  effort  when  tradeoffs  are  made  between  cost,  risk,  performance, 
and  cycle  time. 


Establishing  a  DTC  Metric  at  RMS 


•  RMS  Announced  the  formation  of  the  Engineering  Effectiveness 
Metrics  Council  in  early  2003. 

•  The  Engineering  Effectiveness  Metrics  (EEM)  team  supports  its 
goals  with  three  primary  metrics: 

•  On-Time  Delivery  Performance 

•  First-Presentation  Yield 

•  Design  To  Cost 

•  The  DTC  Metric  is  designed  to  allow  business  unit  management 
to  quickly  review  the  progress  and  ability  of  their  programs. 

•  DTC  Metric  implementation  has  a  phased  approach 

•  SDD  Programs 

•  SDD  and  Production  Programs 

•  CAIV  Metrics 


Implementation 


ayrneon 


#  Potential  programs  taken  from  the  EEM  “Deployment  Matrix,” 
which  are  programs  that  are  reporting  other  EEM  metrics  (First 
Time  Presentation  Yield  and  On-time  Delivery  Performance). 

#  Initial  meetings  with  PLCE  to  identify  candidate  programs. 

#  E-mails  or  phone  calls  to  program  manager. 

#  “Getting  Started”  packets  mailed  to  candidates: 

•  Product  Cost  Control  Survey 

•  DTC  Start-Up  Instructions 

•  DTC  Points  of  Contact 

•  DTC  Guidelines 

•  DTC  Process 

•  Sample  DTC  Plan  Table  of  Contents 

•  CAIV/DTC  Training  Schedules  plus  “Program-Specific”  CAIV/DTC 
offered 


Report  Structure  for  the  DTC  Metric  at  RMS 


9  The  EEM  reporting  organizational  structure  below  is  used  to  facilitate 
executive  level  portfolio  management  of  RMS  programs. 

9  RMS’  Engineering  council  reports  its  metrics  monthly  at  the  Engineering 
level  process  reviews. 


Programs  provide  their  initial  cost  goal  and  current  estimate. 

Programs  provide  an  initial  basis  of  estimate. 

Programs  are  contacted  monthly  for  their  prior  month’s  current  estimate  (trailing 
indicator) 

Current  estimate  is  divided  by  the  cost  goal  for  a  DTC  metric.  This  is  reported  in 
a  percentage  format;  i.e.,  Program  A’s  DTC  metric  is  1 .04,  which  is  4%  over  their 
cost  qoal.  _ 


DTC  Goal  <100%  to  no  >4.99%  over  goal  1 

Over  DTC  Goal  >4.99%  to  9.99% 

Over  DTC  Goal  >10% 

Latency  or  how  often  the  cost  information  is  reviewed  and  updated  is  also 
reported. 


Latency  - 1  -2  months 

Latency  -  2-3  months 

Latency  -  5  months  + 

For  programs  in  “yellow”  and  “red”  categories,  “Root  Causes  and  Corrective 
Actions  reports  are  required. 


DTC  Metric  Definition  and  Reporting  Levels 


Program 

Phase 

Gate 

Metric 

Latency 

Accuracy 

Comment 

One  A 

SDD 

6 

1.12 

1 

+15  -10 

High  Subcontractor  cost  for 
motor  assembly 

Two  B 

SDD 

7 

1.50 

7 

+25  -15 

Program  undergoing  major 
corrections  and  rebaselining 

Three  C 

SDD 

5 

1.04 

4 

+10  -5 

Lower  FPA  Cost 

•  Phase/Gate:  The  program’s  current  position  in  its  life-cycle 

•  DTC  Metric:  Current  Cost  Estimate  /  DTC  Target 

*  Green:  <  1.05;  Yellow:  between  1.05  &  1.1;  Red:  >  1.1 

•  Latency:  Months  since  completion  of  last  Current  Cost  Estimate. 

*  Green:  <  3  months;  Yellow:  between  3  &  5  months;  Red:  >  5  months 

•  Accuracy:  Represents  the  relative  possible  cost  risk  associated  with 
current  cost  estimate  expressed  as  a  plus  and  minus  percent.  Accuracy 
is  not  currently  being  reported. 


DTC  Metric  Reporting  Frequency 
and  Initial  Results 


RMS’s  Engineering  Council  reports  its 
metrics  monthly  at  the  Engineering 
Process  Review  Meeting.  All  engineering 
metrics  are  distilled  into  a  series  of  color- 
coded  stoplight  charts  that  show  current 
status  in  relation  to  goals  for  the  year. 


Variances  are  measured  and  reported  at  design  team 
meetings  and  program  reviews.  Efforts  to  eliminate 
cost  variances  (the  proper  behavior)  become  part  of 
the  IPT  design  effort  when  tradeoffs  are  made 
between  cost,  risk,  performance,  and  cycle  time. 
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Action 

*  “Five  Why’s”  reports  required  for  programs  that  remain  in 
the  red  category  without  signs  of  improvement. 

*  Areas  of  concern  are  identified  for  “yellow”  and  “red” 
programs,  and  Engineering  Centers  assist  in  resolving  the 
challenges. 

*  The  Engineering  Accountability  Review  Group  reviews 
monthly  the  programs  that  should  be  reporting  DTC 
Metrics. 

*  “Programs  That  Should  Be  Reporting  ‘DTC’  Metrics”  are 
now  being  reported  to  Louise  Francesconi  at  the  monthly 
product  line  reviews. 


Lessons  Learned 


•  Affordability  is  the  primary  driver  in  all  architecture  design  and 
development  activities. 

•  DTC  requires  mandatory  cost  requirements  be  assigned  to  all  programs 
down  to  the  lowest  levels. 

•  Programs  must  track  and  measure  their  current  design  to  cost  status 
against  their  goals  at  periodic  intervals.  (Cost  Management) 

•  Cost  must  be  a  design  requirement  with  importance  equal  to  or  greater 
than  performance. 

•  DTC  focus  must  begin  as  early  as  possible  in  a  program  (pre-RFQ)  for 
early  cost  driver  identification. 

•  Cost  estimation  can  be  approximate  in  early  program  phases, 
progressively  better  during  later  phases. 

•  Proper  DTC  behavior  is  achieved  by  setting,  striving  for,  and  ultimately 
reaching  goals.  A  DTC  metric  is  therefore  one  that  keeps  cost  and  cost 
reduction  in  the  forefront  of  IPT  activity. 

•  By  establishing  cost  goals  for  a  program  (and  its  subsystems)  that  are 
time  phased,  and  constantly  decreasing,  a  program  is  able  to  measure  its 
cost  reduction  effort  toward  the  ultimate  program  cost  goal. 


Going  Forward 


To  quote  Sun  Tsu,  The  Art  of  War,  “the  wise  general  in  his  deliberations  must 
consider  both  favourable  and  unfavourable  factors.  By  taking  into  account  the 
favourable  factors,  he  makes  his  plan  feasible;  by  taking  into  account  the 
unfavourable,  he  may  resolve  the  difficulties.” 


Going  Forward:  Plans  for  the  Future 


A  DTC  Metric,  by  itself,  is  not  enough! 

It  is  time  to  consider  expanding  DTC  Metrics  into  CAIV  Metrics: 

•  CAIV  Metrics  encompass  not  only  cost,  but  performance, 
schedule  and  risk  as  well.  The  primary  metric  to  measure 
specific  CAIV  project  effectiveness  is  cost.  The  utilization  of 
this  metric  requires  an  established  cost  baseline  in  sufficient 
detail  to  compare  prior  and  resultant  impacts  of  a  CAIV 
project. 

The  proper  metric  for  CAIV: 

•  Establishes  a  system  cost  goal  for  the  design 

•  Requires  specific  points  of  estimated  development 
production  and  operation/support  costs 

•  Reflects  on  program  costs  and  system  performance 


Going  Forward:  CAIV  Metrics  Sample  Chart 


DTC  Metrics  can  be  enlarged  with  Cost,  Performance,  Schedule 
and  a  Risk  Assessment  to  form  a  set  of  CAIV  Metrics. 


CAIV  Metric 

Threshold 

Goal 

Current 

iCurrent/Goall 

Risk  Assess 

Cost  Driver 

Latency 

Plan  of  Action 

Cost  -  System 

$  32,775.00 

$  31,500.00 

$  37,790.00 

1.20 

Sub-System 

$  5,000.00 

$  4,500.00 

$  6,200.00 

1 .38 

2 

no 

Sub-System 

$  1,500.00 

$  1,500.00 

$  1,400.00 

4 

no 

Sub-System 

$  12,275.00 

$  12,000.00 

$  17,890.00 

1.491 

1 

yes 

Sub-System 

$  8,000.00 

$  7,500.00 

$  6,000.00 

0.80 

3 

yes 

Sub-System 

$  2,500.00 

$  2,500.00 

$  2,700.00 

1.08 

3 

yes 

Sub-System 

3500 

3500 

3600 

1.03 

2 

yes 

Sub-System 

Sub-System 

||  Performance 

Requirement 

Goal 

Current  [ 

|  Req/Current  | 

Risk  Assess 

Cost  Driver 

Latency 

Plan  of  Action  1 

speed  mph 

200 

220 

180 

1 

no 

range  nm 

500 

550 

525 

0.95 

1 

yes 

load  lbs 

750 

750 

800 

0.94 

1 

yes 

KPP-4 

■Schedule 

Contract 

Goal 

Expected 

Exp/Con 

Risk  Assess 

Cost  Driver 

Latency 

Plan  of  Action  I 

Dates 

Months  to  Go 

18 

15 

15 

0.83 

2 

no  1 

Red 

Yellow 

Green 

Blue 

Violet 


Red 

What  is/are 

Red 

Is  there  a  plan  of  action  -  yes/no  1 

Yellow 

the  major  cost 

Yellow 

Comment 

Green 

driver(s) 

Green 

Going  Forward:  CAIV  Metrics  Sample  Chart 


CAIV  Metrics  chart  at  a  glance: 

•  Discloses  a  program’s  status  in  the  areas  of  cost,  performance 
and  schedule.  From  the  above  sample  chart  one  can  quickly  see: 

•  The  program  is  projected  to  over-run  costs  by  20%. 

•  Two  of  the  sub-systems  are  in  the  red;  one  with  a  high  risk  of 
failing. 

•  The  PM  has  no  plan  of  action  to  fix  one  of  the  red  areas 

•  One  sub-system  is  in  the  “violet”  with  low  risk  of  failure  so 
perhaps  cost  goals  ought  to  be  re-allocated. 

•  The  others  are  close  to  goals  on  one-side  or  the  other 

•  Two  of  the  performance  areas  have  superseded  requirements 
while  one  area,  without  a  plan  of  action  and  at  high  risk  of 
failure  is  in  the  red. 

•  And,  the  program  is  planning  on  an  early  delivery. 

•  The  color  coding  helps  management  key  in  on  specific  areas  of 
concern  and  make  necessary  changes. 


Any  Questions? 


•  Now  is  a  good  time  to  ask. 
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Abstract 

Architecting  a  product  requires  a  defined  set  of  requirements  for  the  finished  product; 
e.g.  size,  weight,  volume,  range,  power,  color,  cost,  payload,  etc.  One  very  necessary 
requirement  is  the  anticipated  product  production  cost.  Failure  to  set  the  production  cost 
requirements  at  design  kick-off  allows  for  unexpected  and  unacceptable  production  costs. 
Previously,  all  too  often,  programs  were  allowing  establishment  of  the  production  cost  goal 
to  slip,  or  they  would  wait  for  their  customer  to  establish  it  for  them.  Raytheon  Missile 
Systems’  (RMS)  Engineering  Directorate  has  specified  that  all  development  programs  will 
now  establish  a  production  cost  goal  using  the  Design-To-Cost  (DTC)  metric  described 
within  this  paper  and  will  monitor  their  design  progress  towards  meeting  this  goal.  Each 
program’s  DTC  metric  is  now  collected  monthly  and  reviewed  by  senior  management. 

This  paper  will  focus  on  the  creation  of  the  Design-To-Cost  Metric  (DTC),  its  purpose 
and  use  at  RMS.  The  DTC  metric  is  designed  to  allow  business  unit  management  to  quickly 
review  the  status  of  their  programs  as  to  how  well  the  various  program  designs  are 
progressing  as  to  their  ability  to  be  produced  at  the  specified  value. 

Background 

The  Engineering  Effectiveness  Metrics  initiative  grew  out  of  RMS’s  desire  to  reduce  the 
costs  and  cycle  times  necessary  to  design,  develop  and  build  products  that  work  right  the  first 
time.  As  such,  we  were  challenged  to  measure  factors  that  lead  to  our  products  costing  too 
much,  taking  too  long  to  reach  the  user,  and,  admittedly,  sometimes  not  initially  working 
right.  To  support  these  goals,  an  Engineering  Effectiveness  Metrics  Team  developed  three 
primary  metrics: 

•  On-Time  Delivery  Performance,  which  involves  deliveries  of  Engineering  data, 
documentation,  requirements,  hardware  and  software 

•  First-Presentation  Yield,  which  covers  technical  data  package  change  rates  and 
identifies  out-of-phase  defects  on  all  design,  development  and  production  programs 

•  Design  To  Cost,  which  uses  affordability  principles  to  drive  all  architecture,  design 
and  development  activities. 
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Design  to  Cost 

The  traditional  definition  of  Design  to  Cost  (DTC)  emphasizes  meeting  specific  cost 
targets  at  various  stages  in  a  product’s  life  cycle.  To  meet  customer  affordability  objectives, 
we  must  go  beyond  the  traditional  definition  and  create  a  culture  that  guides  every 
development  and  manufacturing  decision  to  eliminate  non-value  added  activity. 

DTC  is  not  a  stand-alone  process,  and  early  attempts  to  apply  it  as  such  produced 
marginal  results.  DTC  must  be  an  over-arching  philosophy  that  permeates  the  entire 
development  environment  and  dominates  all  major  design  decisions.  DTC  must  treat  cost  as 
a  firm  program  requirement,  applying  the  principals  of  Cost  As  an  Independent  Variable,  or 
as  the  Department  of  Defense  (DoD)  calls  it,  CAIV. 

Design  to  Cost  drives  engineering,  manufacturing,  materiel,  finance,  and  others  to  find 
and  eradicate  extra  cost.  The  DTC  approach  leverages  lean  practices  and  processes  and  uses 
rigorous  cost  target  allocation  together  with  successive  cost/performance  tradeoffs  at  all 
levels  in  the  product  hierarchy  to  achieve  affordable  designs  offering  best  customer  value. 

The  goal  of  the  Design-to-Cost  approach  is  to  meet  production  and  Life  Cycle  Cost 
targets  for  products  that  meet  well- 
established  cost  targets  and  provide 
best  customer  value.  By  integrating 
Design-to-Cost  with  Six  Sigma  quality 
techniques  the  Product  Development 
Process  (PDP)  offers  the  combined 
benefits  of  reduced  production  cost 
and  improved  quality/yield. 

DTC  embraces  and  leverages  the 
concept  of  CAIV.  As  such,  cost  is 
treated  as  a  primary  requirement  that 
may  only  be  traded  for  performance  or 
other  technical  attributes  if  the 
customer  determines  that  doing  so  will 
provide  overall  best  value  and 
establishes  a  new  cost  target  that 
reflects  that  determination. 


DoD  customers  are  focusing  on  affordability  through 
Acquisition  Reform  and  the  best-value  concepts  of  Cost 
As  an  Independent  Variable  (CAIV).  In  response  to  these 
and  related  initiatives,  industry  must  simplify  designs, 
focus  on  best  customer  value,  and  eliminate  non-value- 
added  activity  to  drive  costs  down.  This  objective  is 
possible  if  we,  industry,  focus  on  a  best-value  balance 
among  cost,  performance,  and  supportability  with  the 
same  intensity  that  we  once  devoted  to  performance  alone. 

To  provide  our  customers  with  superior  products  at 
affordable  cost,  we  must  embed  intensive  cost  and  quality 
focuses  into  the  product  development  process  (PDP).  Cost 
focus  needs  to  be  formalized  and  institutionalized  as  an 
integral  part  of  the  PDP.  Cost  assessment  and 
management  begin  long  before  the  product  takes  on  a 
specific  form  and  continue  iteration  with  supportability, 
performance,  and  risk  until  a  best-value  balanced  design  is 
achieved.  (HAC  DTC  Handbook  1996) 
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DTC  and  CAIV 

In  MIL-STD-337,  DOD  once  defined  DTC  as  follows: 

“Design  to  Cost  is  an  acquisition  management  technique  to  achieve  defense  system  designs  that  meet 

stated  cost  requirements.  Cost  is  addressed  on  a  continuing  basis  as  part  of  a  systems  development  and 

production  process.  The  technique  embodies  early  establishment  of  realistic  but  rigorous  cost  targets 

and  a  determined  effort  to  achieve  them.” 

In  more  recent  documentation  such  as  DOD  Instruction  5000.2,  DTC  can  not  be  found, 
but  is  inferred  to  be  under  CAIV.  Over  the  years,  this  has  caused  some  confusion  and  a 
blending  of  terms.  To  clarify  somewhat  and  to  try  to  keep  CAIV  and  DTC  clear: 

•  CAIV  is  architectural  in  nature  and  asks  the  question:  “Given  a  fixed  budget,  how 
much  performance  can  I  get  when  I  need  it  with  maximum  acceptable  risk?” 

•  While  DTC  is  engineering  oriented  and  attempts  to  solve  the  mystery:  “Given  the 
budget  and  performance  requirements,  what  do  I  design  and  build?” 

CAIV  evolved  from  and  expanded  on  the  basic  DTC  concept.  DTC  was,  and  still  is, 
concerned  primarily  with  production  costs  and  was  focused  on  the  lowest  cost  solution 
within  a  given  requirements/performance  set.  CAIV  is  interested  in  costs  from  the  total 
ownership  point  of  view.  The  acquisition  community,  including  technology  and  logistics,  and 
the  requirements  community  uses  the  CAIV  process  to  develop  total  ownership  cost  (TOC), 
schedule,  and  performance  thresholds  and  objectives.  They  address  cost  in  Operational 
Requirements  Documents  (ORDs),  and  balance  mission  needs  with  projected  out-year 
resources,  taking  into  account  anticipated  process  improvements  in  both  DOD  and  defense 
industries.  Thus,  one  can  see  how  DTC  became  a  part  of  CAIV. 

CAIV  and  DTC  at  RMS 

At  RMS,  CAIV  and  DTC  are  blended  into  both  the  business  culture  and  the  development 
process  under  the  heading  of  Affordability.  To  work  effectively,  CAIV  must  be  a  part  of  the 
business  culture,  seeking  cost  reduction  and  best  value  at  every  turn,  while  DTC  must  be  an 
integral  part  of  the  design  and  development  process.  Within  this  process  at  RMS,  well- 
defined  cost  targets  are  assigned  to  each  sub-product  in  a  product’s  hierarchy;  cost  drivers  are 
identified  and  focused  on;  cost/performance  tradeoffs  that  lead  to  affordable,  best-value 
solutions  are  conducted;  and  metrics  are  determined  and  reported  on. 

With  this  approach,  each  design  choice  is  evaluated  simultaneously  for  both  cost  and 
benefit.  This  process  begins  before  Concept  Exploration  and  remains  vigorous  throughout 
product  development.  Focus  is  on  minimizing  cost,  identifying  and  eliminating  non-value- 
added  activity,  reducing  cost  risk,  and  achieving  well-defined  product  cost  objectives  by 
optimizing  the  entire  product  to  provide  best  customer  value.  Affordability  is  interactive, 
creating  a  multi-tier  Virtual  IPT  that  integrates  design  engineering,  process  engineering, 
manufacturing,  materiel,  quality,  and  business  across  all  IPT  levels  to  develop 


Page  3  of  16 


manufacturable  products  that  meet  well-defined  cost  targets  while  balancing  cost, 
performance,  supportability,  and  risk. 


DTC  Process 

DTC,  when  taken  in  its  most  basic  form,  is  very  simple,  concentrating  only  on  meeting 
well-defined  cost  objectives.  In  the  real  world,  many  requirements  must  be  satisfied 
simultaneously,  requiring  thorough  analysis  and  tradeoff  among  viable  alternatives  through 
an  iterative  design  process  to  achieve  a  balanced  affordable  design. 

The  iterative  design  process  begins  at  the  top  level  of  Figure  1  (HAC  DTC  Handbook 
1996)  with  a  product  requirement  that  includes  cost  as  a  major  priority.  The  ensuing 
requirements  flowdown  and  associated  design  process  are  merged  into  a  continuous  multi¬ 
tiered  interactive  process  that  seeks  to  optimize  the  entire  product  by  allocating  all 
requirements  at  all  levels  in  the  product  hierarchy  to  produce  best  customer  value. 
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The  generic  seven  steps  shown  in  Figure  2  (HAC  DTC  Handbook  p.  4-6)  summarize  a 
single  tier  of  the  iterative  decomposition  of  a  product  into  its  sub-products.  This 
representation  of  the  iterative  development  process  is  referred  to  as  the  “Seven  Steps  to  an 
Affordable  Design”. 


The  engineer  must  use  the  following  steps  to 
execute  DTC: 

•  Understand  Requirements  (Cost  Goal  is  a  key 
requirement) 

•  Analyze  Functions 

•  ID  Physical  Alternatives/ Allocate 
Requirements/Plan  Task 

•  Design  Synthesis 

•  Cost  Modeling  -  Estimation  &  Rollup 

•  Evaluate  -  Meet  or  change  Requirements? 

•  Select/Formalize  Design 


Plus,  an  added  step  that  is  rarely  mentioned  and 
often  overlooked:  the  engineer  must  document  and 
report  progress  towards  meeting  the  cost  goal. 


A  product-tailored  derivative  of  these  steps  is 
applied  to  every  sub-product  at  each  level  in  the 
hierarchy  beginning  in  the  preconcept  phase  and 
continuing  throughout  detailed  design.  For  multi-tier 
decompositions,  the  seven  steps  are  applied  at  each 
sub-product  level  until  no  further  decomposition  is 
needed,  so  that  initial  design  approaches  and 
assessments  can  be  used  to  steer  each  level  of 
requirements  flowdown  toward  a  product-global 
optimum.  Cost  target  allocations  are  a  crucial  part  of 
this  flowdown  sequence.  The  iterative  development 
process  relies  on  open  and  ongoing  communication  among  all  IPT  disciplines  and  among 
IPTs  at  all  levels  of  the  product  hierarchy.  This  communication  is  a  powerful  benefit  of  IPD 
and  is  essential  to  the  development  process. 

The  design  is  complete  when  the  customer/contractor  team  has  accomplished  the 
following: 

•  Performed  detailed  cost,  performance,  supportability,  and  risk  assessments  that 
indicate  that  all  final  requirements  will  be  met  with  levels  of  cost,  schedule  and 
technical  risk  acceptable  to  both  the  customer  and  the  company. 

•  Allocated  all  requirements  to  NDI  items  or  specific  custom  components. 

•  Completed  the  detailed  design  of  all  custom  components. 
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Figure  2  Seven  Steps  to  an  Affordable  Design 
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•  Successfully  modeled/prototyped  custom  components  and  assemblies  that  can  drive 
cost,  performance,  or  schedule. 

•  Completed  a  thorough  manufacturing  plan  defining  the  approach  to  the  fabrication  or 
procurement  of  all  components  and  the  assembly,  integration,  and  test  of  the  product 
and  each  significant  subproduct. 

•  Complied  with  all  customer  and  company  requirements  for  ILS,  support,  review, 
documentation,  verification,  scheduling,  warranty,  and  the  like. 

Earlier,  we  mentioned  that  DTC  is  engineering  oriented  and  attempts  to  solve  the 
mystery:  “Given  the  budget  and  performance  requirements,  what  do  I  design  and  build?”  If 
it  were  left  up  to  only  the  engineers,  then  DTC  would  be  simply  one  of  many  engineering 
processes  and  largely  ignored,  but  DTC  has  another  role  within  the  CAIV  framework  as  a 
management  control  system  for  production  costs. 

DTC  as  a  Management  Control  System 

Management  control  systems  for  human  systems  are  equivalent  to  electronic  control 
systems  for  electro-mechanical  systems.  They  are  put  in  place  to  direct  the  targeted  activity 
toward  achievement  of  desired  results. 

The  Cybernetic  Paradigm  and  the  Control  Process  is  one  theory  behind  management 
control  systems.  Comparing  the  Cybernetic  Paradigm  and  Control  Process  from  management 
control  system  theory  (Maciariello,  1984)  to  the  DTC  process,  we  can  easy  add  DTC 
meaning  to  the  Control  Process  diagram  (Figure  3).  Further  comparison  yields  Table  1. 
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Cybernetic  Paradigm  and  Control  Process 

DTC  Process 

Set  Goals  and  performance  measures 

Set  AUPC  Goal  as  part  of  DTC  Plan 

Measure  achievement 

Prepare  current  cost  estimate 

Compare  achievement  with  goals 

Current  estimate  vs.  DTC  Goal 

Compute  the  variances  as  the  result  of  the 
preceding  comparison 

Estimate  System  and  subsystem  variances 

Report  the  variances 

Report  $  Delta 

Determine  cause(s)  of  the  variances 

Cost  Drivers,  spec,  risk,  etc. 

Take  action  to  eliminate  the  variances 

Action  Plan:  Changes 

Follow-up  to  ensure  that  the  goals  are  met 

Repeat  at  interval  per  plan 

Table  1.  Applying  the  DTC  Process  to  Management  Control  System  Theory. 


Converting  the  DTC  Process  into  a  management  control  system,  we  arrive  at  the  process 
shown  in  Figure  4  which  brings  us  to  the  topic  of  metrics.  One  of  the  main  functions  of 
proper  management  is  to  monitor  and  control  a  process;  to  ensure  that  proper  decisions  are 
made;  and  to  act  in  a  timely  manner  to  mitigate  risk.  To  do  so,  a  manager  needs  reliable  and 
accurate  information.  Information  comes  in  many  forms  and  metrics,  if  properly  applied,  are 
invaluable  sources  of  information. 

Both,  the  DTC  process  and  management  control  system  theory  require  establishment  of 
goals,  measurement  of  achievement  of  those  goals,  measurement  of  variances  from  the  goals, 
feed  back  actions  to  the  process  to  attempt  to  eliminate  the  variances,  and  follow  up  to  ensure 
that  the  goals  are  met.  We  also  know  from  theories  on  human  behavior  and  organizations 
that  when  individuals  or  organizations  are  measured  then  those  individuals  tend  to  redirect 
their  effort  towards  achieving  a  level  of  performance  within  the  functionally  being  measured 
that  will  be  at  or  above  the  desired  measurement  grading  level.  We  built  this  aspect  of  human 
behavior  into  our  metric  reporting  system  as  the  organizations  being  measured  know  that 
they  are  being  measured  and  what  they  are  being  measured  against  and  that  the  results  of 
these  measurements  will  be  reported  to  executive  management.  For  this  to  be  successfully 

applied  as  it  has  been  at  RMS,  those 
organizations  that  are  being  measured 
must  see  the  proactive  loop  closure 
that  says  “I’m  making  a  positive 
change  to  do  better.”  Beyond 
identifying  shortcomings, 

measurements  should  be  used  to  help 
engineer  centers/  program  IPTs 
replicate  their  successes. 


To  quote  Sun  Tsu,  The  Art  of  War,  “the  wise 
general  in  his  deliberations  must  consider  both 
favourable  and  unfavourable  factors.  By  taking 
into  account  the  favourable  factors,  he  makes  his 
plan  feasible;  by  taking  into  account  the 
unfavourable,  he  may  resolve  the  difficulties.” 
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Figure  4,  The  DTC  Process  in  Terms  of  a  Management  Control  System 


The  purpose  of  any  metric  is  to  drive  proper  behavior  and  the  purpose  of  the  DTC  Metric 
is  no  different.  The  DTC  Metric,  by  providing  information  to  both  the  program  manager  and 
the  program  manager’s  superiors,  drives  proper  behavior.  The  program  manager  is  aware  of 
a  program’s  cost  status  and  can  make  decisions  based  on  that  status.  The  program  manager’s 
superiors  can  use  the  DTC  Metric  as  is  part  of  a  higher  level  monitoring  function  over 
individual  programs,  across  a  line  of  similar  programs,  or  across  a  business  area.  To  be  of 
value,  it  is  required  that  this  metric  be  reported  to  a  level  significantly  above  the  program  in 
order  for  the  feed  back  to  carry  force  of  action  to  the  program.  It  is  this  force  of  action  that 
provides  an  additional  monitor  and  control  function  to  verify  and  drive  the  program  to 
conduct  a  successful  DTC  program. 
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Establishing  a  DTC  Metric  at  RMS 

Raytheon  Missile  Systems’  Engineering  Directorate  needed  to  have  a  few  simple  metrics 
they  could  use  to  track  and  report  development  program  future  design  production  costs, 
quality,  and  timeliness  on  a  number  of  different  programs.  The  Engineering  Effectiveness 
Metrics  (EEM)  initiative  grew  out  of  RMS’  desire  to  reduce  the  costs  and  cycle  times 
necessary  to  design,  develop  and  build  products  that  work  right  the  first  time. 

Early  in  2003,  RMS  announced  the  formation  of  the  Engineering  Effectiveness  Metrics 
Council  and  its  development  of  metrics  designed  to  help  the  directorate  measure  its 
performance,  identify  problem  areas  and  quantify  its  progress  in  eliminating  them.  Selection 
of  the  appropriate  measurements  would  help  focus  future  efforts  towards  meeting  and 
improvements  in  these  measured  areas.  To  support  these  goals,  the  team  developed  three 
primary  metrics: 

•  On-Time  Delivery  Performance,  which  involves  deliveries  of  Engineering  data, 
documentation,  requirements,  hardware  and  software, 

•  First-Presentation  Yield,  which  covers  technical  data  package  change  rates  and  identifies 
out-of-phase  defects  on  all  design,  development  and  production  programs,  and 

•  Design  To  Cost  (DTC),  which  uses  affordability  principles  to  drive  architecture,  design 
and  development  activities 

The  Engineering  Effectiveness  Metrics  reporting  organizational  structure  (Figure  5)  was 
set  up  to  facilitate  executive  level  portfolio  management  of  RMS  programs  for  their 
achievement  of  selected  goals.  RMS’  Engineering  Accountability  Review  Council,  under 
which  the  EEM  group  functions,  reports  its  metrics  monthly  at  the  Engineering  level 
“Accountability”  reviews. 


Figure  5.  DTC  Metric  reporting  Structure 
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DTC  Metric  Definition 


The  DTC  Metric  was  created  to  allow  management  to  quickly  access  a  program’s 
progress  as  to  how  well  its  design,  as  it  stands  today,  meets  the  cost  target  developed  using 
the  DTC  process.  In  order  to  do  this  we  must  take  snap  shots  of  the  design  at  regular 
intervals.  For  RMS’  DTC  Metric,  the  appropriate  interval  was  determined  to  be  monthly. 
For  each  snap  shot,  a  cost  estimate  is  prepared  using  the  same  set  of  Ground  Rules  and 
Assumptions  used  to  create  the  cost  target.  These  estimates  are  then  compared  to  the  target 
and  evaluated.  The  formula  for  the  DTC  metric  is: 

DTC  Metric  =  Current  Cost  Estimate  /  DTC  Target 

Of  note,  the  current  cost  estimate  and  the 
DTC  cost  target  need  to  be  like  in  nature.  The 
definition  allows  for  targets  and  current 
estimates  to  be  defined  appropriately  for  the 
need  of  each  individual  program.  However,  the 
intent  of  and  management’s  goal  in  establishing 
the  DTC  Metric  (unless  customer  intentions 
dictate  otherwise)  was  for  the  DTC  Metric  to 
measure  the  average  unit  production  cost 
(AUPC)  of  the  product.  The  production  cost  is 
defined  as  the  total  manufacturing  recurring  cost 
with  appropriate  overheads  divided  by  the  total 
production  quantity.  Often,  the  DTC  Target 
represents  a  contractual  cost,  although  it  could 
be  set  lower.  Again,  this  would  be  based  on  the 
ground  rules  and  assumptions.  When  the  DTC 
Metric  is  greater  than  one,  the  current  cost  is 
greater  than  the  cost  target  and  is  an  indication 
of  a  program  starting  to  over  run  its  costs. 

The  DTC  Metric  is  then  color-coded  based  on  severity. 

•  Green  is  when  the  DTC  Metric  is  less  than  1.05, 

•  Yellow  is  between  1.05  and  1.099,  and 

•  Red  is  1 . 1 0  or  greater. 

The  color  scale  allows  a  quick  glance  capability  to  the  monthly  DTC  metric  report  to 
ascertain  the  status  of  all  reporting  program  across  RMS  quickly  (Table  2).  The  scale  was  set 
to  allow  early  intervention  should  a  program  tend  toward  cost  over  runs.  If  a  program  reports 
itself  in  the  yellow  or  red,  management  will  start  to  ask  questions:  questions  that  we’ll  leave 
to  the  reader’s  imagination. 


Establishing  a  realistic  DTC  Goal  (or 
Cost  Target)  is  critical  to  the  DTC 
Metric.  There  are  several  methods  that 
can  be  used  and  as  the  program 
requirements  evolve,  the  method  may 
change  causing  the  target  to  also  change. 
All  changes  should  be  documented. 

Establishing  The  DTC  Goal 

1 .  The  Easy  Way  -  The  Customer  establishes  cost  and  performance 
requrements  for  the  product  in  the  form  of  goals  (objectives)  and 
thresholds 

2.  The  Hot  So  Easy  Way  -  Business  Case  Analysis  and  Technology 
Readiness 

■  Target  Pricing  -  Must  sell  for  5 . (Functions? 

■  Profit  Margn  (Capital  recovery?) 

■  Customer  Desires  vs.  Budgets 

■  "  Ouned"  Technd  ogy  (Facil  iti  es?) 

3.  Fall  Out  Buy  In  -  Allocate  Prog-am  Cost  Goals  - 

Sys.  of  Sys.,  Program,  IPT  or  Subsystem,  par 

4.  Technology  H story  -  Similar  Product  Actuals 

5.  ALL  the  Above! 
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Program _ Jan _ Feb  Mar  Apr _ May  Jun _ Jul 


Program  A 

1.33 

1.11 

1.11 

1.1 1 

1.13 

1.13 

1.13 

Program  J 

1.58 

1.58 

1.58 

1.58 

1.58 

1.42 

1.42 

Program  C 

1.1 1 

1.11 

1.09 

1.07 

Program  F 

1.01 

1.01 

1.01 

1.01 

1.01 

1.08 

1.06 

Program  B 

0.99 

Program  D 

0.99 

0.99 

0.99 

0.99 

0.99 

1.04 

1.04 

Program  E 

0.89 

0.89 

0.89 

0.89 

0.89 

0.89 

0.89 

Program  G 

0.69 

0.69 

0.69 

0.69 

0.69 

0.69 

0.69 

Program  H 

0.86 

0.86 

0.86 

0.86 

0.86 

0.86 

0.86 

Program  1 

0.90 

0.63 

0.64 

0.64 

0.64 

0.64 

0.64 

Program  K 

0.94 

0.94 

0.94 

0.94 

0.94 

0.97 

0.97 

Program  L 

0.90 

0.90 

0.90 

0.90 

0.90 

0.90 

0.90 

Table  2.  DTC  Summary  Chart 

Along  the  same  line,  latency  is  also  reported.  Latency  equates  to  the  number  of  months 
since  completion  of  last  the  Current  Cost  Estimate.  While  it  is  desirable  to  do  this  monthly, 
there  are  occasions  when  this  cannot  be  done.  Latency  is  also  color-coded  and  as  before, 
varying  levels  of  interest  follow  each  color. 


•  Green  is  latency  of  2  months  or  less, 

•  Yellow  is  latency  between  2  and  4  months,  and 

•  Red  is  latency  greater  than  4  months  old. 


Accuracy  of  the  cost  estimate  is  also  reported  and  represents  the  relative  possible  cost 
range  (cost  risk)  associated  with  the  current  cost  estimate.  This  is  expressed  as  a  plus  and 
minus  percent.  Phase  and  Gate  represent  where  in  a  program’s  life-cycle  the  program  current 
is  and  room  for  comments  in  the  DTC  report  exists. 


A  hypothetical  example  is  shown  in  Table  3. 


PROGRAM 

Phase 

Gate 

Metric 

Latency 

Accuracy 

Comment 

One  A 

SDD 

3 

■ 

i 

+15  -10 

High  Subcontractor  cost  for 
motor  assembly 

Two  B 

SDD 

3 

■ 

i 

+25  -15 

Program  undergoing  major 
corrections  and  rebaselining. 

Three  C 

SDD 

2 

1.04 

4 

+10-5 

Lower  FPA  Cost 

Table  3:  DTC  Metric  Report  Including  Accuracy 
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DTC  METRIC  Reporting  Frequency  and  Initial  Results 


RMS’s  Engineering  Effectiveness  Metrics  Council  reports  its  metrics  monthly  at  the 
Engineering  Accountability  Reviews  for  the  previous  month  (trailing  indicators).  All 
engineering  metrics  are  distilled  into  a  series  of  color-coded  stoplight  charts  that  show 
current  status  in  relation  to  goals  for  the  year. 

Data  is  collected  during  the  first  week  of  the  month  and  areas  of  concern  are  pinpointed. 
Product  lines,  programs  and  functional 
organizations  are  informed  when  the 
data  indicates  problems  exist.  Process 
reviews  are  held  the  third  week  of  the 
month,  so  those  areas  of  concern  should 
have  enough  time  to  prepare  for  the 
review  and  to  come  up  with  a  proposed 
solution.  If  solutions  are  not  available 
by  review  time,  the  issue  becomes  an 
action  item,  with  follow-up  slated  for  the 
following  month  or  sooner  depending  on 
severity.  When  product  lines  and 
programs  are  involved,  the  council  informs  the  Engineering  Centers’  IPTs  to  find  a  solution. 

Variances  are  measured  and  reported  at  design  team  meetings  and  program  reviews. 
Efforts  to  eliminate  cost  variances  (the  proper  behavior)  become  part  of  the  IPT  design  effort 
when  tradeoffs  are  made  between  cost,  risk,  performance,  and  cycle  time. 

Implementation  and  Evolution  -  Lessons  Learned 

American  companies  have  long  been  accused  by  foreign  partners  of  wanting  to  close 
the  deal  in  a  short  period  of  time,  but  taking  forever  to  implement  the  decisions.  “Americans 
are  quick  to  sign  a  contract  or  make  a  decision.  But  try  to  get  them  to  implement  it — it  takes 
forever.”  (Ouchi)  Institutionalizing  the  DTC  Metrics  at  Raytheon  Missile  Systems  has  met 
with  some  resistance.  Common  excuses  include:  “The  customer  does  not  require  DTC,” 

“We  didn’t  fund  for  DTC,”  “We  don’t  have  time  to  track  DTC  and  make  reports,”  “We 
received  a  waiver  from  doing  DTC,”  “We’ve  never  had  to  do  this  before,”  and  “This  is  a 
very  small  program,  so  we  don’t  need  to  do  DTC.” 

Top  management  at  Raytheon  Missile  Systems  is  now  closely  watching  our  DTC 
metrics  and  placing  heavy  emphasis  on  successfully  capturing  and  reporting  the  DTC 
metrics.  Answers  to  the  above  comments  include:  “While  the  customer  may  not  require  it, 
this  is  an  internal  RMS  management  requirement.”  “Tracking  the  current  estimate  is 
something  that  a  well-managed  program  is  doing  already,  and  it  is  extremely  easy  and  quick 
to  compare  the  current  estimate  to  the  known  cost  goal.”  “We  are  doing  many  things  that  we 
have  never  had  to  do  before  to  shorten  our  cycle  times,  to  assure  the  quality  of  our  products 
at  initial  delivery,  and  to  stay  within  or  under  budget.  These  things  we  do  to  remain 
competitive,  which  in  turn  safeguards  employment  and  positively  impacts  our  bottom  line.” 
“Even  though  this  is  a  very  small  program,  it  is  of  strategic  importance,  and  offers  lucrative 


Six  months  after  the  DTC  Metric  was  put 
in  place,  analysis  shows  substantial 
improvement.  Data  shows  that  65  percent  of 
the  programs  that  the  council  is  tracking  have 
met  their  DTC  goals,  in  contrast  to  only  33 
percent  six  months  earlier,  with  the  trend 
toward  reducing  costs  existing  for  the  majority 
of  programs  that  have  yet  to  reach  their  DTC 
targets. 


Page  12  of  16 


follow-on  contracts.”  With  top  management  emphasis,  programs  are  now  more  willing  to 
report  their  DTC  metrics  (for  design  and  development  programs)  and  are  embracing 
affordability  metrics  (for  production  programs)  as  well. 

Currently,  the  council  is  continuing  to  refine,  systematize  and  institutionalize  its  data- 
gathering  activities.  When  programs  transition  from  “green”  to  “yellow”  and  from  “yellow” 
to  “red,”  they  are  required  to  provide  “Root  Causes/Corrective  Actions”  explanations  at  the 
Engineering  Accountability  Reviews.  As  mentioned  previously,  the  various  RMS 
Engineering  centers  become  involved  to  assist  in  identifying  solutions  to  programs’ 
challenges.  Should  programs  not  exhibit  improvement  right  away,  a  “Five  Whys”  exercise  is 
conducted,  drilling  down  at  least  five  levels  to  assure  that  the  true  root  causes  have  been 
identified  so  that  they  can  be  corrected.  Simply  put,  it’s  about  improvement.  The  Council  is 
also  analyzing  the  data  to  spot  emerging  trends,  assess  their  significance  and  address 
problems  on  a  case-by-case  basis.  Now  that  the  basic  process  is  up  and  running,  we  are 
looking  more  closely  at  long-term,  systemic  obstacles  and  solutions. 

Adopting  the  correct  metrics  and  using  them  appropriately  can  add  tremendous  value  to 
a  program,  allow  program  managers  increased  ability  to  control  their  programs  vice  being 
controlled  by  their  programs,  and  allow  senior  management  an  earlier  opportunity  to  assist 
programs  heading  toward  problems. 

Key  lessons  learned  are  as  follows: 

•  Affordability  is  the  primary  driver  in  all  architecture  design  and  development 
activities. 

•  DTC  requires  mandatory  cost  requirements  be  assigned  to  all  programs  down  to  the 
lowest  levels. 

•  Programs  must  track  and  measure  their  current  design  to  cost  status  against  their 
goals  at  periodic  intervals. 

•  Cost  must  be  an  independent  design  requirement  with  importance  equal  to  or  greater 
than  performance  (i.e.,  the  process  must  address  CAIV  as  its  primary  focus). 

•  DTC  focus  must  begin  as  early  as  possible  in  a  program  (pre-RFQ)  for  early  cost 
driver  identification. 

•  Lean  practices  and  processes  must  be  effectively  leveraged. 

•  Cost  estimation  can  be  approximate  in  early  program  phases,  progressively  better 
during  Engineering  and  Manufacturing  Development  (E&MD). 

•  Cost  estimation  cycle  time  must  be  near  real-time  by  the  detailed  design  phase. 

•  Design,  manufacturing  and  product  life  cycle  cost  data  must  be  readily  accessible. 

•  DTC  tools  must  be  user  friendly  and  accessible  from  the  IPT’s  desktop. 

•  Manufacturing  process  costs  must  be  understood. 

•  DTC  training  deployment,  and  data  collection  must  be  given  high  priority. 

•  Proper  CAIV  behavior  is  achieved  by  setting,  striving  for,  and  ultimately  reaching 
goals.  A  CAIV  metric  is  therefore  one  that  keeps  cost  and  cost  reduction  in  the 
forefront  of  IPT  activity. 

•  By  establishing  cost  and  TOC  goals  for  a  program  (and  its  subsystems)  that  are  time 
phased,  and  constantly  decreasing,  a  program  is  able  to  measure  its  cost  reduction 
effort  toward  the  ultimate  program  cost  goal. 
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A  Path  Forward 

With  the  Design  to  Cost  Metric  in  place  and  positive  results  being  shown,  it  is  time  to 
consider  expanding  DTC  into  CAIV  and  implementing  a  series  of  metrics  more  in  tune  with 
CAIV  principles. 

CAIV  seeks  to  find  the  optimum  balance  between  Cost,  Performance,  Schedule  and  Risk, 
this  a  set  of  CAIV  Metrics  should  encompass  these  areas.  Table  3  shows  how  the  DTC 
Metric  can  be  enlarged  and  enriched  with  Performance,  Schedule  and  a  Risk  Assessment. 


CAIV  Metric 

Threshold 

Goal 

Current 

Current/Goal 

|  Risk  Assess  | 

Cost  Driver 

Latency 

Plan  of  Action 

Cost  -  System 

$  32,775.00 

$  31,500.00 

$  37,790.00 

Sub-System 

$  5,000.00 

$  4,500.00 

$  6,200.00 

2 

no 

Sub-System 

$  1,500.00 

$  1,500.00 

$  1,400.00 

4 

no 

Sub-System 

$  12,275.00 

$  12,000.00 

-4 

CO 

CD 

o 

o 

o 

1 

yes 

Sub-System 

$  8,000.00 

$  7,500.00 

$  6,000.00 

0.80 

3 

yes 

Sub-System 

$  2,500.00 

$  2,500.00 

$  2,700.00 

1.08 

3 

yes 

Sub-System 

3500 

3500 

3600 

1.03 

2 

yes 

Sub-System 

Sub-System 

Performance 

Requirement 

Goal 

Current 

I  Req/Current  | 

|  Risk  Assess  | 

Cost  Driver 

Latency 

Plan  of  Action 

speed  mph 

200 

220 

180 

1 

no 

range  nm 

500 

550 

525 

0.95 

1 

yes 

load  lbs 

750 

750 

800 

0.94 

1 

yes 

KPP-4 

Schedule 

Contract 

Goal 

Expected 

Exp/Con 

Risk  Assess 

Cost  Driver 

Latency 

Plan  of  Action 

Dates 

Months  to  Go 

18 

15 

15 

0.83 

2 

no 

Red 

Red 

What  i s/are 

Red 

Is  there  a  plan  of  action  -  yes/no 

Yellow 

Yellow 

the  major  cost 

Yellow 

Comment 

Green 

Green 

driver(s) 

Green 

Blue 

Violet 

Table  4:  CAIV  Metrics 


As  a  management  enabler,  this  chart,  at  a  glance  would  disclose  a  program’s  status  in  the 
areas  of  cost,  performance  and  schedule.  From  the  above  sample  chart  one  can  quickly  see: 

•  The  program  is  projected  to  over-run  costs  by  20%. 

o  Two  of  the  sub-systems  are  in  the  red;  one  with  a  high  risk  of  failing, 
o  The  PM  has  no  plan  of  action  to  fix  one  of  the  red  areas 
o  One  sub-system  is  in  the  “violet”  with  low  risk  of  failure  so  perhaps  cost 
goals  ought  to  be  re-allocated, 
o  The  others  are  close  to  goals  on  one-side  or  the  other 

•  Two  of  the  performance  areas  have  superseded  requirements  while  one  area,  without 
a  plan  of  action  and  at  high  risk  of  failure  is  in  the  red. 

•  And,  the  program  is  planning  on  an  early  delivery. 

From  the  above  information,  management  can  key  in  on  areas  to  get  involved  with.  If  a 
program,  like  the  example,  is  in  the  “red”  in  cost  but  ahead  of  schedule  and  over-achieving  in 
some  area  of  performance,  then  perhaps  adjustments  can  be  made  to  lower  costs  while  still 
meeting  requirements.  Likewise,  if  a  program  was  under  cost  but  not  quite  meeting  a  key 
performance  parameter  (KPP)  then  that  area  can  be  targeted  appropriately.  The  color  coding 
helps  key  in  on  specific  areas  of  concern. 
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Of  course,  should  a  Program  Manager  not  have  a  plan  of  action  for  an  area  lacking,  then 
his  or  her  management  would  become  aware  of  this  and  can  make  the  appropriate 
recommendations  to  ensure  correction.  One  would  hope  that  at  the  next  level  up,  all  areas 
lacking  would  have  an  action  plan. 

Going  yet  the  next  step,  the  proper  metric  for  CAIV  is  one  that  establishes  a  system  cost 
goal  for  the  design  and  that  requires  attainment  of  estimated  development,  production,  and 
operation  and  support  costs  (Total  Ownership  Cost)  at  specified  points  along  a  program 
timeline.  It  is  still  too  early  for  this  step,  but  not  to  early  to  considering  how  to  incorporate  it 
and  when.  In  addition,  other  metrics  can  be  included  that  reflect  on  program  costs  and 
system  performance  and  can  be  used  by  management  to  enable  an  informed  decision. 
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Top  Five  Systems  Engineering  Issues* 


•  Lack  of  awareness  of  the  importance,  value,  timing, 
accountability,  and  organizational  structure  of  SE  on 
programs 

•  Adequate,  qualified  resources  are  generally  not  available 
within  government  and  industry  for  allocation  on  major 
programs 

•  Insufficient  SE  tools  and  environments  to  effectively 
execute  SE  on  programs 

•  Requirements  definition,  development,  and  management 
is  not  applied  consistently  and  effectively 

•  Poor  initial  program  formulation 


*  Based  on  an  NDIA  Study  in  January  2003 
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Recap:  What  We  Have  Done  To 
Revitalize  Systems  Engineering 


•  Issued  Systems  Engineering  (SE)  policy 

•  Issued  guidance  on  SE  and  Test  &  Evaluation  (T&E) 

•  Integrating  Developmental  T&E  with  SE  policy  and 
assessment  functions  -  focused  on  effective,  early 
engagement  of  both 

•  Instituted  system-level  assessments  in  support  of  OSD 
major  acquisition  program  oversight  role 

•  Established  SE  Forum  -  senior-level  focus  within  DoD 


•  Working  with  Defense  Acquisition  University  to  revise 
SE,  T&E,  and  enabling  career  fields  curricula 

•  Leveraging  close  working  relationships  with  industry  and 
academia 


Necessary  but  not  sufficient! 
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General  Approach:  Program  Outreach 

Review  Products 


Full  reviews  conducted  9-12  months  before  Milestone 

-  Detailed  findings,  risks  &  actionable  recommendations 

-  Conducted  in  “PM  support”  vice  “OSD  oversight”  mode 
“Quick-Look”  reviews  conducted  2-3  months  before  Milestone 

-  Same  form  and  formats  as  full  assessment;  conducted  “for  record” 
review 


•  Quarterly  Defense  Acquisition  Executive  Summary  (DAES) 
assessments  inputs 

•  Test  &  Evaluation  Master  Plan  (TEMP)  and  Systems  Engineering 
Plan  (SEP)  development  and  approval 
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Systems  Engineering  Plans 
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DoD  Systems  Engineering  Shortfalls* 


•  Common  failures  on  acquisition  programs  include: 

-  Inadequate  understanding  of  requirements 

-  Lack  of  systems  engineering  discipline,  authority,  and  resources 

-  Lack  of  technical  planning  and  oversight 

-  Stovepipe  developments  with  late  integration 

-  Lack  of  subject  matter  expertise  at  the  integration  level 

-  Availability  of  systems  integration  facilities 

-  Incomplete,  obsolete,  or  inflexible  architectures 

-  Low  visibility  of  software  risk 

-  Technology  maturity  overestimated 


Major  contributors  to  poor  program  performance 


*  Findings  from  PSRs  and  DoD-directed  Studies/Reviews 
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Systems  Engineering  Plan  Activity 

(since  November  2004) 


•  Number  of  SEPs  reviewed: 

59 

•  Programs  submitting  SEPs: 

36 

u  Number  of  SEPs  approved: 

8 

u  Number  of  SEPs  pending:  5 

•  Reviews  planned  for  rest  of 

FY06:  103 

Component-Managed 

Acquisitions 

Air 

Force 

23%  Other 

20% 


Navy 

31% 
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Army 

26% 


SEP  Program  Milestones 


P re  MSB 
56% 


Pre  MS  C 
25% 


Special 

Interest 

16% 


Pre  MS  A 
3% 


Programs  by  Product  Line 


Rotary  Business 

Wing  -  5  Comms  -  4  Systems  -  5 


Fixed  Wing  - 
5 

Sea 
Systems  -  3 


Land 

Systems  -  4 
C2/ISR  ■  9 


Unmanned 
Systems  -  1 


Emerging  SEP  Comments** 

(not  systemic  across  all  programs) 


Technical 
Review 
Planning 
24%  ' 


Technical 
Baseline 
Management 
Planning 
17%  ’ 


Integration 
with  Overall 
Management  of 
Program 
23% 


Program 

Requirements 

18% 


Technical 
Staffing  and 
Organizational 
Planning 
18%  ’ 


**BASED  ON  ANALYSIS  OF  27  OUT  OF  39  PROGRAMS 


Version  1.0;  CM#  05-10-002-P 


8 


Version  1.0;  CM#  05-10-002-P 


Program  Support 


General  Review  Areas 


ASSESSMENT  METHODOLOGY  FOR  PRE-MILESTONE  C 

Mission  Capabilities/Requirements  Assessment  Area  4 

Sub-Area  1.1  -  Operational  Requirements_ 4 

ASSESSMENT  METHODOLOGY  FOR  PRE-MILESTONE  B 


1.0 


3.0 


2.0 


3.0 


4.0 


4.0 


5.0 


6.0 


5.0 


6.0 


Mission  Capabilities/Requirements  Assessment  Area  4 

Sub-Area  1.1  -  Operational  Requirements _ 4 


1.0 

ASSESSMENT  METHODOLOGY  FOR  PRE-MILESTONE  A 

Mission  Capabilities/Requirements  Assessment  Area 

4 

Sub-Area  1.1  -  Operational  Requirements 

4 

2.0 

Resources  Assessment  Area 

9 

Sub-Area  2.1  -  Program  Planning  and  Allocation 

9 

Sub-Area  2.2  -  Personnel 

10 

Sub-Area  2.3  -  Facilities 

12 

Sub-Area  2.4  -  Engineering  Tools 

13 

3.0 

Management  Assessment  Area 

16 

Sub-Area  3.1  -  Acquisition  Strategy/Process 

16 

Sub-Area  3.2  -  Project  Planning 

19 

Sub-Area  3.3  -  Program  and  Project  Management 

21 

Sub-Area  3.4  -  Contracting  and  Subcontracting 

26 

Sub-Area  3.5  -  Communication 

28 

4.0 

Technical  Process  Assessment  Area 

30 

Sub-Area  4.1  -  Technology  Assessment  and  Transition 

30 

Sub-Area  4.2  -  Requirements  Development 

31 

Sub-Area  4.3  -  Functional  Analysis  &  Allocation 

32 

Sub-Area  4.4  -  Design  Synthesis 

33 

Sub-Area  4.5  -  System  Integration,  Test  and  Verification 

35 

Sub-Area  4.6  -  Transition  to  Deployment 

37 

Sub-Area  4.7  -  Process  Improvement 

38 

5.0 

Technical  Product  Assessment  Area 

38 

Sub-Area  5.1  -  System  Description 

38 

Sub- Area  5.2  -  System  Performance 

42 

Sub- Area  5.3  -  System  Attributes 

43 

6.0 

Environment  Assessment  Area 

44 

Sub-Area  6.1  -  Statutory  and  Regulatory  Environment 

45 
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Program  Support  Review  Activity 

(since  March  2004) 


Number  of  PSRs  completed:  25 
Number  of  AOTRs  completed:  4 

Reviews  planned  for  rest  of  FY06 
“  PSRs:  at  least  24 
«  AOTRs:  2 


Service-Managed  Acquisitions 


Agencies 


Army 

16% 


Marine 

Corps 

8% 
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Navy 

20% 


Reviews  Conducted  Prior  to  Each 
Milestone 


Pre-MS  C 
16% 


Other 

32% 


Pre-MS  B 
40% 


Pre-FRP 

8% 


Pre-MSA 

4% 


Programs  by  Product  Line 


Rotary- 

Wing 

Aircraft  16% 


Space 

Systems 


Business 
Systems  4% 


Fixed-Wing 
Aircraft  32% 


Land 

Systems  8% 
C2/ISR  12% 


Unmanned 
Systems  4% 
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Samples  of  Program  Support  Review  “Strengths” 


•  Experienced  and  dedicated  program  office  teams 

•  Strong  teaming  between  prime  contractors,  sub-contractors, 
program  offices  and  engineering  support 

•  Use  of  well  defined  and  disciplined  SE  processes 

•  Proactive  use  of  independent  review  teams 

•  Successful  management  of  external  interfaces 

•  Corporate  commitment  to  process  improvement 

•  Appropriate  focus  on  performance-based  logistics 

•  Notable  manufacturing  processes 

•  Focus  on  DoD  initiatives 

•  Excellent  risk  management  practices 


But  not  on  all  Programs. . . 
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Emerging  Program  Support  Findings** 

(not  systemic  across  all  programs) 


Findings  across  the  6  general  review  areas... 

(based  on  assessment  methodology  areas) 


Technical 

Product 

20% 


Technical 

Process 

20% 


Environment 

4% 

Mission 
Capabilities 
12% 


Management 

24% 


Resources 

20% 


**BASED  ON  ANALYSIS  OF  14  OUT  OF  22  REVIEWS 
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Driving  Technical  Rigor  Back  Into  Programs 
“How  PMs  are  reacting  to  PSR  recommendations?” 


•  Mission  Capabilities  -  Requirements 

-  User  requirements  not  fully  defined  and/or  in  flux 

a  Established  requirements  management  plan  with  all  stake  holders,  including  proactive 
plan  for  Net-Ready  KPP 

•  Resources  -  Personnel 

-  Experienced,  dedicated  PM  office  staff,  but  stretched  too  thin 

a  Expanded,  empowered  WIPT  to  bring  in  technical  authority  SMEs,  users,  and  DCMA 

•  Management  -  Schedule  Adequacy 

-  Technical  review  planning  demonstrated  schedule  was  high  risk 

a  Lengthen  schedule  to  include  full  suite  of  SE  technical  reviews,  supported  by  adjusted 
program  funding 

•  Technical  Process  -  Test  &  Evaluation 

-  Insufficient  reliability  growth  program  to  meet  user  requirements  by  IOT&E 
a  Increased  the  number  of  test  articles  and  added  sub-system  level  test  events 

•  Technical  Product  -  S u ppo rtab i I i ty/M ai ntai n ab i I ity 

-  Logistics  demonstration  plan  just  prior  to  IOT&E 
a  Demonstration  re-scheduled  prior  to  MS  C 
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Better  than  90%  acceptance  of  recommendations 


14 


Systemic  Analysis  Perspective 


ow  do  we  find  solutions  to  the  systemic  problems? 


55 


PSR 

Findings 


Program 

Unique 

Causes 


Program 
Unique 
Recommen¬ 
dations^ 


PSR  \  \  \ 

Systemic  Analysis 


Systemic 

Issues 


Root 

Causes 


Systemic 

Solutions 


* 


DoD  Acquisition  ^ 
Community 


•  Policy/Guidance 

•  Other  Processes  (JCIDS,  etc) 

•  Education  &  Training 

•  Oversight  (DABS/ITAB) 

•  Best  Practices 

•  Execution  (staffing) 
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Number  and  Type  of  Findings  by  Program 


160 


140 


120 


</> 

100 


"O 

c 


80 


2 

E 

3  60 


40 


20 


G  H  I  J  K  L  M  N 

Programs  *  Data  from  14  Program  Support  Reviews 


□  1.1  ■  2.1  □  2.2  □  2.3  ■  2.4  □  3.1  □  3.2  □  3.3  ■  3.4  □  3.5  □  4.1  □  4.2  ■  4.3  ■  4.4  □  4.5  ■  4.6  □  4.7  □  5.1  □  5.2  □  5.3  □  6.1 


Numbers  represent  sections  of  the  PSR  Metholodogy 
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Systemic  Analysis  Perspective 


“What  are  the  systemic  problem  areas?” 

Systems  Engineering 
Test  &  Evaluation 
Maintainability 
Software 

Integration/Interoperability 
Requirements 
Schedule 

0  2  4  6  8  10  12  14 

Number  of  Programs  Where  Issue  Was  Prevalent 

□  Pre-MS  B  ■  Pre-MS  C  □  Pre-FRP 
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Representative  Issues 

(1  of  3) 


•  Representative  Issues  for  Schedule 


-  Schedules  too  aggressive 

-  Detailed  schedules  missing  key  components 

-  Schedule  concurrency  (e.g.  T&E  activities) 


•  Representative  Issues  for  Requirements 

-  Requirements  don’t  support  planned  modifications,  increasing  capacity 

-  Requirements  changed  without  consideration  or  coordination  with 
PM/PO  and  dependent  programs 

-  “Shortsighted”  requirements,  i.e.  safety  critical,  bandwidth  to  support 
future  capabilities 

•  Representative  Issues  for  Intearation/lnteroperabilitv 

-  Integration  plans  lacking  key  components 

-  Multi-platform,  scalable  design  benefits  not  realized  due  to  low  hw/sw 
commonality 

-  Interoperability  with  Joint  Forces  not  adequately  addressed 
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Representative  Issues 

(2  of  3) 


•  Representative  Issues  for  Software 

-  Software  processes  not  institutionalized 

-  Software  development  planning  doesn’t  adequately  capture  lessons 
learned  to  incorporate  into  successive  builds 

-  Systems  and  spiral  software  requirements  undefined 

-  Software  architecture  immature 

-  Software  reuse  strategies  are  inconsistent  across  programs 

-  Software  support  plan  missing 


•  Representative  Issues  for  Maintainability 

-  Maintainability  requirements  incomplete  or  missing 

-  Diagnostic  effectiveness  measures  are  either  too  ambiguous  or  missing 

-  Tailoring  out  of  criticality  calculations  translates  to  inability  to  monitor  the 
maintainability  status  of  reliability  critical  items 
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Representative  Issues 

(3  of  3) 


•  Representative  Issues  for  Test  and  Evaluation 

-  No  reliability  details  (hours,  profile,  exit  criteria,  confidence  level,  OC 
curve) 

-  Lack  metrics 

-  Basis  for  some  threat-based  requirements  not  fully  explained  or 
rationalized 


•  Representative  Issues  for  Systems  Engineering 

-  Lack  of  disciplined  SE  process,  metrics,  etc 

-  PO  not  conducting  PRR  prior  to  LRIP 

-  Missing  Joint  CONOPs 

-  Missing  System  Functional  Review  (SFR)  and  PDR  during  SDD 
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•  We  are  working  to  meet  the  Under  Secretary's 
imperatives  in  support  of  transformation  by: 

-  Providing  a  context  for  decisions 

-  Putting  credibility  into  the  acquisition  process 

-  Driving  systems  engineering  back  into  programs 

•  Our  ultimate  goal  in  conducting  PSRs  is  to  help  all 
programs  achieve  mission  success  through: 

-  Early  and  persistent  application  of  SE 

-  Event-driven  technical  reviews  and  test  programs 
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Questions... perhaps  Answers 
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Raytheon 

Customer  Success  Is  Our  Mission 


C  Ml  for  services 


Gordon  Ward  Director  Of  Quality  &  R6s™  Raytheon  RIS 
Juan  Cava,  Mark  Pumar,  Enterprise  Process  Group  Raytheon 


Agenda 


Raytheon 


•  Background 

•  Lessons  Learned 

•  History 

•  Epiphanies 

•  Approach 

•  Conclusion 


“...he  who  attempts  it  must  first  pass  the  point  of  this 
lance; "  and  so  saying  he  brandished  it  so  stoutly  and 
dexterously  that  he  overawed  all  who  did  not  know  him. 

Miguel  de  Cervantes 
Don  Quixote 
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Background  -  Critical  Issues 


Raytheon 


•  Need  to  be  CM  Ml  (L3)  to  stay  in  Business. 

•  Service  Organization  (that  don't  fit  into  the  traditional  product 
development  model)  struggle  to  achieve  CM  Ml  L3  in  a  timely 
and  cost  effective  manner. 

•  How  does  an  organization  staffed  with  practitioners  from 
standard  (product  oriented)  high  maturity  organizations,  with 
large  range  in  its  technical  disciplines,  little  or  no  process 
dollars,  and  little  or  no  project  autonomy  achieve  a  CMMI 
level  3? 

•  Welcome  to  the  World  of  Technical  Services! 
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Background  -  Pasadena  Operations 


Raytheon 


•  Raytheon’s  Pasadena  Operations 

-  Part  of  Raytheon  Information  Solutions. 

-  Establish  in  1998  with  the  award  of  the  SDSIO  (Science  Data  System 
Implementation  and  Operations)  contract  by  the  California  Institute  of 
Technology  and  NASA’s  Jet  Propulsion  Laboratory  (JPL). 

-  Umbrella  services  contract  allowing  JPL  Managers  to  contract  directly  with 
Raytheon  for  technical  and  scientific  services. 

-  Consists  of  functional  departments  organized 
along  lines  of  businesses  reporting  to  a 
Program  Manager. 

-  Technical  disciplines  range  from  software  and 
systems  engineering  to  IT,  scientific  analysis, 
and  web  development. 
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Background  -  Challenges  to  CMMI 


Raytheon 


•  Service,  not  product-based  organization. 

-  No  traditional  end-to-end  lifecycle. 

-  Customer  directing  work  -  little  or  no  autonomy. 

-  Small  overhead  -  no  funding  for  process  support  &  development. 

•  Blurring  of  function  within  projects  and  across  departments. 

-  Project  activities  range  from  software  development  to  graphic  design. 

-  Departments  support  operations,  IT,  analysis,  and  development. 

•  Raytheon  and  Customer  culture  significantly  different. 

-  Research  versus  product  oriented. 

-  Low  process  maturity  (relative  to  industry). 

-  Small  teams  (1-2  FTE)  with  modest  funding. 
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Process  Improvement  History 


Raytheon 


•  Grass  root  effort 

-  Started  with  (unfunded)  special  interest  group  in  March  1999. 

-  Prevailing  feeling  that  Customer  would  be  better  served  if  proven 
process  could  be  applied  to  each  task. 

-  Most  members  worked  for  high  maturity  organizations  before  coming  to 
Pasadena. 

•  False  starts  and  dead  ends  1 998  -  2003 

-  Top-down  approach  to  process  improvement  (unsuccessful) 

-  Obstacles: 

•  CMM,  no  CMMI  yet 

•  Lack  of  process  infrastructure  (QA,  CM,  MA,  etc.) 

•  Customer  “owns”  project  areas  (PP,  PMC,  IPM,  Risk) 

•  Small,  diverse,  short-term  (  <  year)  projects 

•  Customer  chooses  not  to  perform  some  key  practices 

•  Funding 
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Process  Improvement  History  (cont.) 


Raytheon 


•  Alternatives 

-  Approach:  Restrict  process  to  one  or  two  projects  where  sufficient 
autonomy  existed  to  apply  Model. 

-  Drawbacks:  No  benefit  on  most  projects;  limited  benefit  to  customer. 

-  Approach :  Apply  Model  to  a  collective  group  of  projects,  with  no  single 
project  performing  all  key  practices  of  the  Model. 

-  Drawbacks:  Risky  -  might  loose  one  or  more  key  projects. 

-  Approach:  Completely  new  (non-traditional)  approach. 

-  Drawbacks:  Not  clear  how  to  proceed;  requires  ‘breakthroughs . 
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History  (cont.) 


Raytheon 


•  Breakthroughs  2003  and  2004 

-  Use  bottom-up  approach. 

•  Develop  lifecycle  based  on  how  work  is  actually  done. 

•  Map  Model  to  lifecycle. 

•  “Fill”  process  gaps. 


-  Shift  focus  to  delivery  of  service  instead  of  delivery  of  a  product. 

-  Use  Raytheon  Six-Sigma  for  process  improvement. 

Raylheon  Six  Sigmai' 

-  Use  a  traditional  engineering  lifecycle  (requirements,  design, 
implementation,  verification  and  validation)  to  develop  the  process. 


-  Employ  an  evolutionary  or  staged  approach  to  implementation. 
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Epiphanies 


Raytheon 


•  Key  breakthroughs  that  resulted  in  substantial  progress  in 
applying  the  CMMI  to  services. 

•  Epiphany  1 :  Task  Orders  are  equivalent  to  projects. 

-  Apply  process  to  every  task  order. 

-  Re-identify  each  Model  key  practice  with  an  equivalent  practices  in  the 
service  environment. 

•  Epiphany  2:  Project  requirements  are  services  requested  by 
the  Customer. 

-  Requirements  are  the  Customer  requests  for  services  in  the  form  of 
personnel  and  attendant  support. 
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Epiphanies  (cont.) 


Raytheon 


•  Epiphany  3:  Every  project  has  the  same  (unchanging)  five 
requirements. 

-  Requirements  are: 

•  Staffing  -  e.g.  supply  two  oceanographers. 

•  Facilities  -  e.g.  provide  office  space  and  equipment  for  assigned  personnel. 

•  Finances  -  e.g.  monitor  and  report  cost  associated  with  supplying  2 
oceanographers. 

•  Management  -  e.g.  manage  the  task  order  (find  staff  and  facilities,  monitor 
cost  and  personnel). 

•  Infrastructure  Support  -  e.g.  provide  networks,  phones,  computers,  etc.  for 
2  oceanographers. 
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Epiphanies  (cont.) 


Raytheon 


•  Epiphany  4:  The  relative  time  spent  for  development  versus 
delivery  in  services  is  reversed  from  that  of  products. 

-  Products:  most  of  the  effort  is  spent  developing  the  product. 

-  Services:  most  of  the  effort  is  spent  delivering  the  service. 


Product 

Development 

i 

Deliver 

Service 

(Development 

Deliver 

_ i 
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Approach 


Raytheon 


•  Template-based  solutions. 

-  Technical  staff  and  management  not  burdened  with 
process  details  that  are  not  directly  applicable  to 
their  work. 
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—  Technical  staff  works  with  solutions  that  are 
relevant  to  their  everyday  tasks  without  having  to 
become  versed  in  the  CMMI. 

-  A  relatively  small  group  of  Model  experts  can 
concentrate  on  insuring  that  the  CMMI  practice 
areas  are  covered  via  the  usage  of  the  templates. 
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Approach  (cont.) 


Approach  Used 

•  Multi-part  SCAMPI  C  and  B. 

—  SCAMPI  C  divided  into  two  events. 

•  1  st  event  examined  organization’s 
business  model. 

•  2nd  event  examined  templates. 

-  SCAMPI  B  divided  into  three  events. 

•  1 st  event  examined  evidence  from  the 
project  area  of  the  Model  on  a  single 
focus  project. 

•  2nd  event  examined  evidence  from  the 
support  area  of  the  Model. 

•  3rd  event  examined  evidence  from  the 
engineering  area  of  the  Model. 

-  Engaged  appraisal  team  in 

improvement  process 

•  Team  recommendations,  solutions,  and 
feedback  incorporated  into  process  before 
deployment. 

-  SCAMPI  A:  Traditional 


Raytheon 


Traditional  Approach 

•C  and  B  SCAMPIs  are  conducted 
as  single  events 

—SCAMPI  C  reviews  policies  & 
procedures 

-SCAMPI  B  reviews  polices, 
procedures  and  artifacts 


Page  13 


Lessons  Learned 


Raytheon 


Use  Bottoms-up  Approach 

-  Develop  process  solutions  based  on  business  model 

-  Tailor  solutions  to  the  organization 


Run  the  implementation  as  a  (serious)  project 

-  Establish  a  project  manager,  budget,  schedule,  and  measurable  goals. 

-  Track  and  monitor  progress  on  a  regular  basis  using  EVMS. 

-  Use  phased  deployment 

-  Develop  and  validate  processes  before  deployment. 

Implement  a  ‘ grass  roots’  communications  plan  throughout  the 
project 

-  Start  communication  with  staff  and  management  early  to  establish  and  clarify 
goals. 

-  Celebrate  small  successes  publicly  at  all-hands  meetings 
and  other  group  events. 

-  Setup  recurring  open  houses  and  training  sessions 
with  the  process  developers. 

-  Demonstrate  the  benefits  to  individuals. 


.'J  Countdown  tg 

SCAMPI  A 


Lewi  3  Certification! 
1 0- NOV-2004 
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Lessons  Learned  (cont.) 


Raytheon 


*  Obtain  stakeholders’  support  and  active  involvement. 

-  Gain  sponsors  at  the  highest  level  and  understand  their  goals. 

-  Involve  Customers  frequently  via  EPG  and  Steering  Committee 
meetings. 

-  Communicate  the  benefit  of  reaching  goals. 

•  Make  use  of  consultants. 

-  Leverage  Model  expertise  from  other  parts  of  the  organization. 

-  Choose  the  lead  appraiser  wisely  i.e.  ‘out-of-box’  thinker. 

-  Put  the  appraisal  team  to  work  for  the  organization. 

-  Use  their  feedback  to  refine  processes  before  deployment. 
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Conclusions 


RayHieon 


•  Successfully  implemented  one  of  the  first 
applications  of  the  CMMI  Model  to  Services  _ 

-  Technical  Details  Documented  in  (up-coming)  -sa,  fs.'rn.  «7-  as-®!. 

SEI  Technical  Note  '  '  "  * 

-  https ://bscw.sei.cmu.edu/pub/bscw.cqi/0/79783  'm* 


•  Developed  pragmatic  and  cost  effective 
model  to  deploy  CMMI  in  non-traditional 
applications 

•  True  Process  Improvement 

-  Added  processes  meaningful  and  useful  to 
organization 

•  Day-to-day  operations  improved 

•  Organization  more  efficient  and  effective  at 
delivering  services 

-  Increased  engagement  with  customer  at  all 
levels 
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SYSTEMS 

ENGINEERING  AND 
THE  SOFTWARE 


.AW5  O 


M/ll  M 


omas  F.  Christian  Jr. 
Director  of  Engineering 
ACSSW,  WPAFB  OH 

Oct.  2005  — = 


SYSTEMS  ENGINEERING 


*  Systems  Engineering  is  the 
disciplined  application  of  tools  and 
principles  to  achieve  a  complex  goal 

*  Software  Engineering  is  part  of 
Systems  Engineering 

*  Systems  Engineering  must  obey  the 
Fundamental  Laws  of  Physics 


THE  LAWS  OF  PHYSICS 

But  what  are  the  fundamental  laws 
of  physics? 


F=MA? 


Earlier  seminal  work  at  SSTC  2005 
said  “Yes”  -  Newton’s  Laws  of 
Motion  govern  Software  Engineering 


THE  LAWS  OF  METAPHYSICS 


•  “I  think  therefore  I  code” 

•  No,  No!! 

•  Even  EEs  understand  Inertia  but, 
could  there  be  some  law  even  more 
fundamental  than  that?  _  _ 


E  LAWS  OF  NATI 
PHILOSOPHY 


-  YES  - 


THERMO 


The  unexplainable 


am 


1 ST  LAW  -  CONSERVATION 
OF  ENERGY 

Ao 


Heat  Transfer 
Ao  c  AQ 

AS  =  Entropy  =■ — j— 


(COLD) 


“If  the  state  of  a  system  is  changed 
by  applying  work  or  heat  or  both, 
then  the  change  in  the  energy  of  the 
system  must  equal  the  energy 

applied” 
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2nd  LAW  -  TENDENCY 
TOWARD  EQUILIBRIUM 

“It  is  impossible  to  move  heat,  by 
cyclical  process  from  something  at 
lower  temperature  to  something  at 
higher  temperature  unless  work  is 
added  to  the  system” 

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 

i  i  i  i  i  i  i  i 


3rd  LAW  -  ABSOLUTE  ZERO 


“If  the  entropy  of  each  element  at 
absolute  zero  can  be  taken  as  zero, 
then  all  elements  above  absolute  zero 
must  have  a  finite,  positive  entropy; 
however,  because  entropy  cannot  be 
reduced  to  zero  by  finite  means  (as  per 
the  Second  Law),  no  system  can  reach 

absolute  zero” 


”Since  we  are  Systems  Engineers 
-  Not  Physicists  - 
Let’s  put  them  into  System 
Engineering  -  Speak 

•1st  Law:  You  Can’t  Win  -  Just 

Break  Even 

•  2nd  Law:  You  Can  Only  Break  Even 

at  Absolute  Zero 

3rd  Law:  You  Can’t  Reach  Absolute 

Zero 
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THE  SOFTWARE  LAWS  OF 
THERMODYNAMICS 

*  Optimizing  software  quality,  cost, 
schedule  require  proper  processes, 
planning,  and  people 

*  Proper  processes,  planning  and 
people  requires  time  to  do  it  right 


There  is  NEVER  time  to  do  it  right 


“RASSA’S  LAW” 

“No  one  can  resist  the  temptation  to 
edit  another’s  work  or  start  coding 
on  the  first  day  of  a  program” 


Robert  M.  Cranwell,  Manager 
Readiness  &  Supportability  Programs 

Sandia  National  Laboratories  (SNL) 
Albuquerque,  NM  87185 


8th  Annual  NDIA  Systems  Engineering  Conference 

October  24-27,  2005 
San  Diego,  CA 


Phone:  (505)844-8368  Fax:  (505)844-3321 
Email:  rmcranw@sandia.gov 
Web  Site:  reliability.sandia.gov 

ndia  is  a  multiprogram  laboratory  operated  by  Sandia  Corporation,  a  Lockheed  Martin  Company, 
for  the  United  States  Department  of  Energy  under  contract  DE-AC04-94AL85000. 


New  Mexico 


■  8,000  employees  ill  New  Mexico, 
California,  Nevada,  and  Hawaii 

■  Responsible  for  research,  development, 
engineering,  and  maintenance  of  U.S. 
nuclear  weapons 


San  Jfe  Noil  Mial  Labciatnries 


“ Helping  secure  a  peaceful  and 
free  world  through  technology  ” 


Sandia  National 
Laboratories 


■  Responsible  for  all  non-nuclear 
subsystems;  the  primary  systems 
integration  and  engineering  lab 

■  $2B  annual  budget 

■  S550M  from  other  federal  agencies 

■  $50M  from  private  industry  through 
R&D  partnerships 


California 


<r**R 


Major  Readiness/Supportability  Initiatives 


Army’s  FCS  Supportability 

FJJTURE  COMBAT  SYSTEMS 


ure 

Combat 
Systems 


(izrArZBli 


<e  Atmy/MRPA/htdosity 


JSF  Autonomic  Logisti 


Prognostics  & 

HIGHLY-RELIABLE  Health  Management 
AIRCRAFT 

I  A. 


TECHI® 

3j\I/\BL3 


Joint  Strike  Fighter 

Autonomic  Logistics  System 


-OGICALLY 


AUTONOMIC  LOGISTICS 
INFORMATION  SYSTEM 


s  Recapitalization  (RECAP) 

t».uh  Miitj.ij  ■M^bihiii  ftMrii  ihmminu} 


Navy  CVN  21  Manpower 


Sandia 

Matronal 

laboratories 


e 

ombdt 

Sustems 
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Distribution  authorized  to  the  Department  of  Defense  and  U.S.  DoD  contractors  only  by  direction  of  PM  UA,  August  2004. 

Other  requests  shall  be  referred  to  PM  UA. 


What  is  a  System-of-Systems? 

^  ^  FUTURE  CC 


FUTURE  COMBAT  SYSTEMS 


—A- 

*  -  M 

-  ^ 

(he  learn  -  The  Amy/DARPA/hdushy 

Key  to  understanding  a  System-of-Systems  (SoS)  is  the 
notion  that  a  SoS  performs  a  function  not  possible  with 
any  of  the  individual  parts  (systems)  acting  alone 


•  In  this  context,  a  SoS  can  be  viewed  as  a  collection  of  interdependent 
systems  that  are  integrated  to  provide  a  prescribed  capability 

•  The  loss  of  individual  systems  within  the  SoS  will  degrade  the 
performance  or  capabilities  of  the  SoS 

•  However,  individual  systems  within  the  SoS  can  provide  a  capability 
or  function  independent  of  the  other  systems  within  the  SoS 


A  SoS  can  be  viewed  as  multiple  systems,  each  capable  of  independent 
operations,  but  must  interact  in  order  to  fulfill  a  global  mission. 


@0 


Sandia 

National 

Laboratories 


SNL  SoS  M&S  Framework 


FUTURE  COMBAT  SYSTEMS 


State  Models  l 


•  Single  model  for  multiple  MOEs/MOPs 

•  Multiple  states  (not  just  functional  or  failed) 

•  Incorporates  complex  interdependencies 

•  Includes  external  factors  (weather,  terrain,  threats) 

•  Able  to  incrementally  add  new  components 
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State  Model  Objects 


Optimization 

Analyses 


Uncertainty  Characterization 


A 


n 


Simulated  Campaigns 

*  Scalability  to  large  numbers  of  platforms 

*  Can  include  multiple  pulses  throughout  campaigns 

*  Includes  integrated  logistics  support  during  campaign 

*  Tracks  dynamic  changes  in  force  structure  during  campaign 

*  Includes  complex  communications  &  data  networks 


O^Tsm-lheAnnY/MRPA/htdusiry 

-  Unique  State  Modeling  Concepts 

-  State-of-Art  SoS  Simulation  Framework 

-  Customized  Optimization  Algorithms 

Provides  Capability  for  Analyzing: 

-  Systems  as  well  as  Systems-of-Systems 

-  Warfight  capabilities  (pre,  post,  during  campaigns) 

-  Spiral  integration  of  technologies 

Flexibility  to  Include: 

-  Complex  supply  &  support  networks 

-  Stochastic  treatment  of  combat  damage 

-  Human  performance  (maintainers,  warfighters, ...) 

Allows  Analyst  to  Assess  Impacts  of: 

-  PHM 

_ Asset  visibility 

Commonality  &  complex  functional  redundancies 


Time-Dependent  Warfight  Capability 
UA/SoS  MOE’s/TPM’s 
Sensitivity  Trades 
Optimization  Analyses 


Sandia 

Matronal 

Laboratories 


Integrated  methodology  provides  unique  capabilities  for  SoS  analysis 


State  Model  Concepts 


•  State  modeling  attributes 


FUTURE  COMBAT  SYSTEMS 
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-  Multiple  functions/operations 

♦  Mobility 

♦  Communications 

♦  Sensing 

♦  Lethality/Firepower 

-  Multiple  States  (not  just  functional  or  failed) 

-  Models  interdependencies 

-  Can  include  external  factors  (weather,  terrain,  combat, ...) 

-  Able  to  incrementally  add  new  components 

•  Model  system  behavior  by  defining: 

-  States  for  all  subsystems/components/functions 

-  How  transitions  are  made  between  states 

•  States  can  change  through: 

-  Normal  processes  (failure,  repair,...) 

-  External  conditions  (weather,  terrain,  combat,...) 

-  Changes  in  functional  states  of  other  systems 


NLOS-C  State  Model  Object 


Example  Elements 


•105  mm  Cannon 
•M240  Machine  Gun 
•Sandstorm 

Example  Operations 
•Operability 
•Lethality 
•Mobility 


Sandia 

Matronal 

Laboratories 


Intuitive  Environment  for  Capturing  System  Behavior 


Soldier  Performance  Modelin 


Multiple  Functions/Operation 

-  Navigate 

-  Communicate 

-  Combat 

-  Maintain  . . . 

Multiple  States 

-  Not  just  functional  or  failed 

Model  soldier  behavior  by  defining: 


Mobility 

Lethality 

Survivability 

Communication 


|  Perceptive  Systems 


Systems  Engineering 
Infrastructure 


Cognitive  Systems 


Episodic  Memory 


-  All  possible  states 

-  How  transitions  are  made  between  states 

•  States  can  change  through: 

-  Fatigue 

-  Stress 

-  Sleep  deprivation 

-  Training  . . . 


Human  Performance  is  Key  to  Modeling  Complex  SoS’s 


Sandia 

National 

Laboratories 


State  Model  Results 


FUTURE  COMBAT  SYSTEMS 


System  Performance  Analyses 

-  Reliability 


n  r  rr.  .  J  //  I 


System  Reliability 


15 


12 


-  MTBF 

-  Availability ... 

•  Sensitivity  Analyses 

•  Optimization  Analyses 

-  Spares 

-  Performance  trades 

-  Resource  allocation 


Rank: 


Export.. 


1  Summary 


2  Details 


Performance  Measure 

Baseline  Limit  Objective 

Value 

Fitness 

0.004G1 

0.7581 

NLOS-C :  Reliability 

0.535  0.580  0.800 

0.584 

C2 :  Reliability 

0.773  0.300  0.820 

0.807 

Cost  1 

0  2.00E+7  1.20E+7 

1.23E+7 

<  1  > 

0.75  0.76  0.77  0.78 

Reliability 


0.79 


0.80 


Contributors  to  System  Failure 


2  3 

Failure  Modes 


1:  Diesel 
Engine 

2:  Alternator 


3:  Battery 


4:  FCS 

Electrical 

Sys. 


<  I  »».] 


Sandia 

Matronal 

Laboratories 


Time  Simulation 


FUTURE  COMBAT  SYSTEMS 
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5  ? 
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Incorporates  State  Model  Objects  (SMOs)  to  dynamically 
represent  a  given  scenario  for  a  defined  force  structure 


Creates  and  duplicates  platforms  and  platform  types  (Defines  Force  Structure) 
Describes  Scenario/Campaign 
Describes  Functions  for  Each  Platform  Type 
Describes  System/Elements  Properties 
Describes  External  Conditions 


•  Evaluates/tracks  Functionality  of  Force  Structure  during  Campaign 

•  Tracks  Supplies  &  Services  (Logistics  Footprint) 


Sandia 

Matronal 

Laboratories 


Defining  Force  Structure 


FUTURE  COMBAT  SYSTEMS 


_  g  — ^  __  mm  m  t  T^angThe Amty/mPA/hxkatry 

Example  Battalion  Structure  for  Simulation 


Simulation  Input 


Structure  j  Assign  Systems  j 


E  Battalion 
E  Company  A 
Platoon  1 
Platoon  2 
E  Company  B 
Platoon  1 
Platoon  2 


Add 


E  Battalion 
E-  Company  A 
-  Platoon  1 

NLOS-C-O01 
ICV-001 
RSV-001 
NLOS-C-002 
NLOS-C-003 
NLOS-C-O04 
NLOS-C-005 
NLOS-C-006 
NLOS-C-O07 
NLOS-C-008 
C2V-004 
ICV-002 

i  n\  {  nn^i 


Save 


d 


Cancel 


HHC  BIC  CAB1  CAB2  CAB3  AV  Sqd  NLOS  Batt  FSB 


HHC  Recon  MCS1  MCS2  INF1  INF2  NLOS  M 


HQ 


PLT1 


PLT2 


PLT3 


C2 


Troop  Carriers 


Mules  W/GST 


Troop  Carriers 


SUGV 


UAV1-AV 


Force  Structure  is  established  in  Simulation  Software  by  creating 

and  duplicating  platforms  and  platform  types  gH 
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Simulation  Scenario  Structure 


Edit  Scenarios 


Scenarios 


Ground  Vehicle  Scenario 


iL 


End  On 

Length 

Direction 

Speed 

Location 

Desired  Stat  Condition  ^ 

1 

Time  jJ 

72 

Field 

Operating 

None  — 1 

2 

Distance 

“H24 

Repair  Facili 

Operable 

None 

3 

Time 

■72 

Field 

Operating 

None 

All  States  True  | 

Edit  Scenarios 


Scenarios 


Air  Vehicle  Scenario 


Ground 

Adi 


Air  Vehicle  Scenario 


“3 


Systems 


Length 

'irectio 

Speed  Location 

Desired  State 

Condition 

Level  ^  1 

1 

2 

Field 

Operating 

IDBMjJ 

o—l 

2 

G 

Repair  Facility 

Operable 

None 

0 

3 

2 

Field 

Operating 

None 

0 

4 

S 

Repair  Facility 

Operable 

None 

0 

5 

2 

Field 

Operating 

T  urbulence 

1 

6 

G 

Repair  Facility 

Operable 

None 

0 

7 

2 

Field 

Operating 

T  urbulence 

1 

8 

6 

Repair  Facility 

Operable 

None 

0 

9 

2 

Field 

Operating 

None 

0 

10 

6 

Repair  Facility 

Operable 

None 

0 

11 

2 

Field 

Operating 

None 

0 

d 

LsJ 

_ 

Ll 

Add 


Delete 


Copy 


Rename 


Apply 


FUTURE  COMBAT  SYSTEMS 


'  RPA/kuhtstry 


Ground  Vehicle 
Ground  Vehicle 
Ground  Vehicle 
Ground  Vehicle 
Ground  Vehicle 
Ground  Vehicle 


Scena 

Scena 

Scena 

Scena 

Scena 

Scena 


3 


Systems 


UAV-020 

:  Air 

Vehicle 

Scenario 

UAV-021 

:  Air 

Vehicle 

Scenario 

UAV-022 

:  Air 

Vehicle 

Scenario 

UAV-023 

:  Air 

Vehicle 

Scenario 

UAV-024 

:  Air 

Vehicle 

Scenario 

UAV-025 

:  Air 

Vehicle 

Scenario 

UAV-026 

:  Air 

Vehicle 

Scenario 

UAV-027 

:  Air 

Vehicle 

Scenario 

UAV-028 

:  Air 

Vehicle 

Scenario 

UAV-029 

:  Air 

Vehicle 

Scenario 

UAV-030 

:  Air 

Vehicle 

Scenario 

UAV-031 

:  Air 

Vehicle 

Scenario 

UAV-032 

:  Air 

Vehicle 

Scenario 

3 


Select  All 
P  By  Type 


Select 


H 


C  By  Scenario  [Ground  Vehicle  Scenario 


laboratories 


unctions  for  Each  Platform  Type 


FdA  Funt  Liunv 


^2Y-UQ1 


CZv-D02 

CZ^-DOE 
C?/  COE 

tCV-Otfl 

icv-aoc 

»CV-0D3 
!CV-00i 

PID/-ODG 


z  cf'r.n  p_| 

Mobile 

rtf  ho 


A 


FdiE  himtboni 


Fninctrona 


RSV-010 
RSV-Q11 
R5V-CJ12; 
RSV-D1  3 
RSV-014 
RSV-015 
RSV-dl? 
UAVtJOl 

HESSES 

UAV-QD3 

UAVUU4 

IJAV-QrF. 

UAV-OK 


Moblrii. 


C3 

Sensing 


=S  DcjpUP  |  O 


Add 


WAT  SYSTEMS 


Che  km  -Ike  Afitty/BARPA/kx&tstry 


Functiniw- 


Delete 


Copy 


t-4 

zJ 


General  ijiHsets  Hucceis  Hatha  | 


Function  ID: 


Uci  enphorr 


System  Stale  M  Yeilowr 
Operabla 

■  Ino-p^-Mhle 


3y?hn.ii  Rl-i-l—  1 1  Rh-1 
(  rtpefabie 

^  Inoperable 


Rcn-sme  |  i0  a  syaie  m  tc  s-d  t  its  iu  net  ops  . 


TFTT  '"^uikh 

w  laboratories 


Example  External  Conditions 
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Che  Jean  -The  Amy/DARFA/hdushf 


Edit  Conditions 


Conditions  |  Apply  To  |  T riggers  | 
Conditions 


[Turbulence 


Copy 


Rename 


External  Conditions 


Number  of  Ranges 


P  i 


Apply  To 


f  All  Systems 


System  Type  |nL0S-C 


Selected  System 


Elements 


i 


105  mm  Cannon 
Alternator 


Axlel 
Axle  2 
Axle  3 
Axle  4 

Diesel  Engine 
FBCB2  Network 
Fire  Control 
FLash  Detector 
FUR  Imaging 


|  Fuel  System 


Glint  Detector 
Instrumentation 


M240  Machine  Gun 
MGV  Batteries 
MGV  Elec.  System 


NBC  Sensor 
SINCGARS  Radio 


hi 


Edit  Element  Modification  Factors 


Select  one  or  more  elements  that  the  external  condition  will  affect. 

Done 

Cancel 

External  Conditions  Can  Include: 

•Terrain 

^Rough 
^MOUT . . . 

•Weather 

^Sandstorm 
^Turbulence . . 

•Combat . . . 

External  Conditions  Affect: 

•Consumables 
^Water/Fuel 
s Ammo . . . 

•Random  Processes 

^Time-to-failure  (TTF) 
^Time-to-repair  (TTR) . . . 

•System/Element  State(s) 
s  Operable 
^Inoperable . . . 


Flexibility  in  Treatment  of  External  Conditions 
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System  Functional  Status 


FUTURE  COMBAT  SYSTEMS 
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System  Functional  Status 


Current  Results 

Systems 

Element  States 

|  Availability  | 

FUTURE  COMBAT  SYSTEMS 
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States  by  System  Type 


Name  State 

Time-in-State 

Expected  TIS 

Age  Accel. 

i 

1 

Diesel  Engine 

True 

292.85  1,536.84 

1.00 

2 

Fuel  System 

True 

303.65 

4,671.61 

1.00 

3 

Instrumentation 

True 

301.26 

7,481.27 

1.00 

4 

MGV  Batteries 

True 

292.65 

2,715.61 

1.00 

5 

MGV  Elec.  System 

True 

296.57 

2,651.20 

1.00 

6 

Steering  System 

True 

304.51 

2,616.73 

1.00 

7 

Suspension 

True 

302.47 

6,135.14 

1.00 

8 

T  ransfer  Case 

True 

300.05 

5,732.35 

1.00 

9 

T  ransmission 

False 

0.00 

- 

1.00 

10 

Axle  1 

True 

306.62 

6,277.27 

1.00 

11 

Axle  2 

True 

293.22 

6,022.82 

1.00 

12 

Axle  3 

True 

303.70 

5,866.13 

1.00 

13 

Axle  4 

True 

291.08 

6,255.95 

1.00 

14 

Wheel  1L 

True 

293.05 

2,347.93 

1.00 

d 

Name 

Capacity 

Remaining 

Projected  Time 

Request  Replenish 

1 

Fuel 

100.00 

0.00 

0.00 

True 

2 

Water 

20.00 

0.00 

0.00 

True 

d 

Close 
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FCS  UA/BCT  Analyses 

Example  FCS  UA/BCT  Analysis 

•  BCT  force  structure  consisting  of  1552  platforms 

•  Multiple  pulses 

•  Level-5  subsystems  -  on  average  265  elements 

•  Performed  optimization  and  sensitivity  analyses 
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FCS  Equipped  Brigade  Combat  Team  (BCT) 


Intelligence, 
Networking  & 
Sensing 


HQ 


2014  Increment  1  Threshold 
102  Infantry  Carrier  Vehicle 
49  Command  and  Control  Vehicle  C2V 
30  R&SV 

60  Mounted  Combat  Systems 
24  NLOS  Mortar 
18  NLOS  Cannon 

29  FCS  Medical  Vehicle  ( 10  MV-T  / 19  MV-E  ) 

10  FRMV 


Sustainment 


Analysis  & 
Processing 


108  UAV  CL  I  Aerial  Vehicles 
36  UAV  CL  II  Aerial  Vehicles 
48  UAV  CL  III  Aerial  Vehicles 
8  UAV  CL  IVa  Aerial  Vehicle 
16  UAV  CL  IVb  Aerial  Vehicle 
216 


■Class.lVUAV 


Combined  Arms  Battalion 

1 


Reconnaissance 
Surveillance  &  Target 
Acquisition 


27  ARV-RSTA 
18  ARVA(L) 

18  ARV-Assault 
54  MULES-T 
60  NLOS  LS 
81  SUGV 
30  MULES-CM 
6  MULES- RT 

544  Trucks 


FSB 


Sustainment 


Maintenance 


][ 


Medical 


Class  I  UAV 


Fire  Support 


Infantry 

Company 


Mortar 

-Battery 


•  xt 


Class  I  &  III  UAVs 


Class  I  &  II  UAVs 


Infantry 

Platoons 


(SrsL 


Developers, 


FCS  Common  Modeling  Framework 

FUTURE  COMBAT  SYSTEMS 


PM  Unit  of  Action  (UA)  Common  Supportability  M&S  Framework 


Sustainment 

Planning 


Demands 


Optimal 

Strategies 


Alternatives 


Platform  data 


Sensitivities 


FoS  Metrics 


Sandia  SoS  Models 


Drivers 


Optimal 

Allocations 


Common  Toolset  creates 
Unifying  Point  of  Reference 

•  Facilitates  Communication 

•  Increases  Understanding 

•  Avoids  Surprises 


] 


SNL  tools  being  incorporated  into  PM  UA  Common  M&S 
Framework  for  use  in  SoS  Supportability  Assessment  Modeling 


Sandia 

National 

Laboratories 


Support  Enterprise  Model  (SEM)  a 


A  Unique  Logistics  Modeling,  Analysis  and  Decision  Support 

^^1.  - 


Forward 

Supply 


Features 

-  Integrated  modeling  of  worldwide  support  system 

♦  Operations 

♦  Supply/Repair  Chain 

♦  Transportation 

-  Dynamic  changes  throughout  life-cycle 

♦  Inventory/Fleet  build  up  and  retirement 
S' Site  activation/closure 

♦  Deployment/surge 

-  Total  support  system  performance  &  cost 

♦  Full  on/off-system  support  activities 

♦  Prognostics  and  Health  Management  (PHM) 

♦  Global  optimization  across  enterprise 

Benefits 

-  Real-time  strategic  planning  support 

-  Dramatic  risk  mitigation 


Support  System  Analysis 

Integration  of  Complex  &  Dynamic  Support  Processes 


Operating  Stes 

U 

j&SBte- 


Regional 

Supply 


la 


Transportation 


Operations  at  Sea 


Dynamics  of  Support 

•  A/C  Deliveries  &  Site  Activation 

•  Deployment  &  Surge  Operations 

•  Autonomic  Logistics  Decisions 
Performance  and  Cost 

•  Aircraft  Availability  &  Sorties 

•  Spares  &  Personnel  Quantities 

•  Performance  vs.  Cost  Trades 


Supply  Deployed 
Repair  C^erationS 


Repair  Supply 


Deployed 

Supply 


Unparalleled  resource  management  flexibility  to  deal  with  changing  conditions 


SEM  is  a  Development  Design  Tool  and  a  Decision/Planning  &  Management  Tool 


Sandia 

National 

Laboratories 


Typical  Problem  Scale  (JSF) 


3,000  aircraft 

-  5000+  LRC’S 

-  10  configurations 


Repair 


250 
500  types 


50  personnel  s 


30  -  50  years  of  simulation 


REi 


~4M  data  elements 


»  1 M  part  demands 
Arbitrary  multi-echelon  support  structure 


CONUS 


OCONUS 


Carrier 


Unprecedented  Data  Integration  Challenge 


Sandia 

National 

Laboratories 


SEM  Optimization  Approach 


Unique  hybrid  optimization  process  for  SEM 

Heuristics: 

Analytic  Mixed  Integer  Programming: 

Solution  Refinement 

Problem-Oriented  Factors 

Unprecedented  problem  scale 

♦  Roughly  1  million  site/part  combinations 

♦  Roughly  20K  site/resource  combinations 

♦  Per  year  of  the  simulation 

♦  On  the  order  of  50  million  variables  in  all 
SEM  simulation  is  stochastic 

Algorithmic  Challenges 


Simultaneous  optimization  of  inventory  levels,  resource  levels,  and  location  of  repair  facilities 

♦  Most  approaches  only  consider  one  of  these  facets 
Avoiding  “brittle”  solutions 

♦  Algorithms  must  generate  robust  solutions  -  insensitive  to  minor  changes  in  operational 
parameters 

♦  Need  to  characterize  the  trade-offs  between  solution  robustness  and  cost,  quantify  risk 


Goal:  Arrange  spare  parts,  support  equipment  and  personnel  skills 
across  the  enterprise  to  meet  target  performance  at  lowest  cost 


BAE  SYSTEMS 


Industry  Perspectives  and  Identified  Barriers 
to  the  Use  of  MIL-STD-882D  for  integrating 
ESOH  Considerations  into  Systems 


Jon  Derickson,  PE,  CSP 

Manager,  FCS  Environmental,  Safety  and  Health 
Ground  Systems  Division 
BAE  Systems,  Land  &  Armaments 


October  26th  2005 


Jon  Derickson  -  ESOH  Manager,  BAE  Systems  -  Ground  Systems  Division 
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Outline 


BAE  SYSTEMS 


■  Systems  and  Safety  Engineering 

■  MIL-STD-882D 

■  Potential  Barriers  for  Integrating  Safety  into  Systems 
Engineering 
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Systems  and  Safety  Engineering 


BAE  SYSTEMS 


■  Systems  Engineering  -  The  design  of  a  complex  interrelation  of  many 
elements  (a  system)  to  maximize  an  agreed  upon  measure  of  system 
performance,  taking  into  consideration  all  of  the  elements  related  in  any  way  to 
the  system,  including  utilization  of  worker  power  as  well  as  the  characteristics  of 
each  of  the  system's  components 

■  System  Safety  -  The  optimum  degree  of  safety  within  the  constraints  of 
operational  effectiveness,  time  and  cost,  attained  through  specific  application  of 
system  safety  engineering  throughout  all  phases  of  a  system 

■  System  Safety  Engineering  -  An  element  of  systems  management 
involving  the  application  of  scientific  and  engineering  principles  for  the  timely 
identification  of  hazards  and  initiation  of  those  actions  necessary  to  prevent  or 
control  hazards  within  the  system 


Reference:  McGraw-Hill  Dictionary  of  Scientific  and  Technical  Terms 
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One  Aspect  of  Systems  Engineering 


BAE  SYSTEMS 


Systems  Engineering 

System  Safety  Engineering 


Reliability  Engineering 


Maintainability 
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MIL-STD-882D  Objectives 


BAE  SYSTEMS 


■  The  DoD  is  committed  to  protecting: 

►  private  and  public  personnel  from  accidental  death,  injury,  or  occupational 
illness 

►  weapon  systems,  equipment,  material,  and  facilities  from  accidental 
destruction  or  damage 

►  public  property  while  executing  its  mission  of  national  defense. 

■  Within  mission  requirements,  the  DoD  will  ensure  that  the  quality 
of  the  environment  is  protected  to  the  maximum  extent  practical. 

■  The  DoD  has  implemented  environmental,  safety,  and  health 
efforts  to  meet  these  objectives.  Integral  to  these  efforts  is  the  use 
of  a  system  safety  approach  to  manage  the  risk  of  mishaps 
associated  with  DoD  operations. 

■  A  key  objective  of  the  DoD  system  safety  approach  is  to  include 
mishap  risk  management  consistent  with  mission  requirements,  in 
technology  development  by  design  for  DoD  systems,  subsystems, 
equipment,  facilities,  and  their  interfaces  and  operation. 

■  The  DoD  goal  is  zero  mishaps. 


October  26th  2005 


Jon  Derickson  -  ESOH  Manager,  BAE  Systems  -  Ground  Systems  Division 


5 


Major  Elements  of  MIL-STD-882D 


BAE  SYSTEMS 


■  Document  your  approach 

■  Identify  Hazards 

■  Assess  Risk 

■  Identify  hazard  risk  mitigations  (requirements) 

■  Reduce  the  risks  to  acceptable  levels 

■  Verify  risk  reduction 

■  Formally  obtain  approval  of  residual  risk 

■  Conduct  hazard  tracking  of  hazards  and  risk 
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System  Safety  Objectives 


BAE  SYSTEMS 


“The  principle  objective  of  a  system  safety  program  within  the  DoD  is  to  make 
sure  safety,  consistent  with  mission  requirements,  is  included  in  technology 
development  and  designed  into  systems”  (Ref  MIL-STD-882C) 

■  Safety,  consistent  with  mission  requirements  is  designed  into  the 
system 

■  Hazards  are  identified,  tracked,  evaluated,  and  eliminated  or  the  risk  is 
reduced  to  acceptable  levels 

■  Historical  safety  data  is  considered  and  used  in  new  designs 

■  Changes  in  design  or  mission  requirements  maintain  an  acceptable 
level  of  risk 

■  Significant  safety  data  is  documented  as  “lessons  learned”  for  future 
development 
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Overall  System  Safety  Process 


BAE  SYSTEMS 


f  \ 

Understand 
the  system 


•  Intended  use 

•  Foreseeable 
Misuse 

•  Operational 
Environments 

•  Operator 
Interface 

•  Maintenance 

•  Testing 

•  Training 

•  Shipping 

•  Storage 


4 


4 


4 


4 


Identify  Hazards 


Evaluate/ Assess  the  risks 


Verify  Implementation 


Obtain  Approval  of  Residual  Risk 


October  26th  2005 


Jon  Derickson  -  ESOH  Manager,  BAE  Systems  -  Ground  Systems  Division 


8 


Precedence  for  Hazard  Mitigations 


BAE  SYSTEMS 


r - : - ^ 

Always  do  #1  first 

1)  Design  to  Eliminate  Hazards 

2)  Incorporate  Safety  Devices 

3)  Provide  Warning  Indicators 

4)  Develop  Procedures,  Warnings,  and  Training 


Work  down  the  list  only  after  previous  items  have  proven  not  effective 


Combinations  of  these  controls  are  used 
to  develop  “System”  of  hazard  controls 
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Potential  Barriers  -  Speed  Bumps 


BAE  SYSTEMS 


■  Systems  engineers  are  not  always  familiar  with  System 
Safety  approaches,  terms  and  processes 

■  System  safety  engineers  are  not  always  familiar  with 
systems  engineering  approaches,  terms  and  processes 

■  Timing 

►  Safety  often  gets  involved  too  late  in  the  process 

►  They  don’t  always  know  about  key  systems  deadlines 

►  They  often  are  asked  for  inputs  at  the  end  -  during  document 
release  process 
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Potential  Barriers  -  Jersey  Barriers 


BAE  SYSTEMS 


■  Process  Barriers 

■  Tools  Barriers 

■  Rainy  Days  and  Sunny  Days 
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Process  Barriers 


BAE  SYSTEMS 


■  Systems  engineering  processes  change  significantly  from 
program  to  program 

►  Safety  engineering  needs  to  be  involved  in  the  definition  of 
processes  in  addition  to  definition  of  safety  requirements  and 
constraints 

■  Bait  and  switch 

►  Everyone  agrees  on  one  process  and  then  the  process  is  over¬ 
taken  by  program  schedules  and  other  events 

■  Hazard  discovery  often  continues  through  system 
development  so  key  requirements  sometimes  come  late 

■  System  safety  is  not  a  Key  Process  area  for  CMMI  so  it 
often  gets  left  out  of  systems  engineering  processes 
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Tools  Barriers 


BAE  SYSTEMS 


■  How  to  integrate  hazard  tracking  into  systems  tools  (UML, 
RTM,  DOORS,  etc) 

►  Not  always  effective  to  implement  tracking  in  the  systems  tool 

►  This  needs  to  be  thought  through  early  in  a  program  to  ensure 
effective  implementation  and  to  avoid  false  starts 

■  Different  disciplines  of  systems  engineering  use  different 
tools  and/or  different  documents  for  different  parts  of  a 
system 

►  This  often  makes  it  difficult  to  fully  specify  hazard  controls  that 
thread  through  an  entire  system 

►  We  end  up  developing  a  safety  view  of  mitigation  approaches 
that  captures  hardware,  software,  and  functional  aspects  of  the 
system 
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Rainy  Days  and  Sunny  Days 


BAE  SYSTEMS 


■  Systems  engineers  are  often  concerned  with  what  we 
WANT  the  system  to  do  (Sunny  Day  Scenario’s) 

►  They  strive  for  clear  concise  requirements  that  are  verifiable 

■  Safety  engineers  are  often  concerned  with  what  we 
DON’T  want  the  system  to  do  (Rainy  Day  Scenario’s) 

►  Accidents  are  often  caused  by  a  string  of  unlikely  events 

►  The  combinations  and  permutations  of  possibilities  become 
very  complicated  and  is  very  difficult  to  clearly  specify  in 
neat  little  requirements  at  the  beginning  of  a  program 

■  Need  to  find  Effective  ways  to  capture  what  we  Don’t 
Want  in  a  way  that  can  be  verified  and  validated 


Jon  Derickson  -  ESOH  Manager,  BAE  Systems  -  Ground  Systems  Division 
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Summary 


BAE  SYSTEMS 


■  Recognize  that  system  safety  is  part  of  systems 
engineering  and  sometimes  plays  a  key  role 

►  Cultivate  and  nurture  a  cooperative  environment  between 
system  safety  and  systems  engineering 

►  Make  sure  that  system  safety  is  integrated  in  the  systems 
processes 

►  Know  when  you  need  to  get  safety  involved 

■  Make  sure  system  safety  is  involved  early  in  both 
requirements  development  and  process  development 

►  Make  sure  they  are  aware  of  your  processes  and  key 
deadlines 

■  Look  for  the  win-win  opportunities 
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BAE  SYSTEMS 


BACKUP  SLIDES 
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System  Safety  Approach 


BAE  SYSTEMS 
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Hazard  Control  Development 


BAE  SYSTEMS 


Hazard 

Scenario 

General 

Approach 

Risk 

Assess 

for  Controlling 

Identified 

Hazard 

Comments 

Hazard  Scenario 
&  Risk  Assessment 


Hardware  Requirement  1 

Hardware  Requiremen 

X 

Software  Requirement  1 


Software  Requirement 


Procedural  Controls  1 


Procedural  Controls  X 


-  Description  of  Concern 

-  Effects  on  People  &  Equipment 

-  Risk  Assessment 

Probability 

Severity 

Risk  Assessment  Code 

-  Background  Information 


Hazard  Controls 

-  Design  Approach 

-  Software  Requirements 

-  Hardware  Requirements 

-  Interface  Requirements 

-  Warnings  and  Cautions 

-  Procedures 


Documented  in  Hazard  Documented  in  Hazard  Tracking  System 

Tracking  System  and  Software  Safety  Tracking  System 
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Cost  of  Safety  Changes 


BAE  SYSTEMS 


Concept  Specification  Design  Validation  Production  Fielded  Operations 
Development 

Life  Cycle  Phase 
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Risk  Assessment  Criteria 


Category 

Level 

Description 

Catastrophic 

1 

Event  results  in  death,  permanent  total  disability,  loss  of  FCS  assets  exceeding  $1M,  or  irreversible 
severe  environment  damage  that  violates  law  or  regulation  and/or  FCS  Program  stoppage. 

Critical 

II 

Event  results  in  permanent  partial  disability,  injuries  or  occupational  illness  that  may  result  in 
hospitalization  of  >  5  days,  loss  of  FCS  assets  exceeding  $200K  but  less  than  $1 M,  or  a  reversible 
environment  damage  causing  a  violation  of  law/regulation,  or  a  FCS  Program  delay. 

Marginal 

III 

Event  results  in  injury  or  occupational  illness  resulting  in  hospitalization  of  <  5  days,  loss  exceeding  $40K 
but  less  than  $200K,  or  mitigatible  environment  damage  without  violation  of  law  or  regulation  where 
restoration  activities  can  be  accomplished. 

Negligible 

IV 

Event  results  in  injury  or  illness  not  resulting  in  hospitalization  of  <  1  day,  loss  exceeding  $2K  but  less 
than  $40K,  or  minimal  environment  damage  not  violating  law  or  regulation. 

Qualitative  Description 

Level 

Likelihood 

Individual  Item 

Fleet  or  Inventory 

A 

Frequent 

Likely  to  occur  often  in  the  life  of  an  item,  with  a  probability  of  occurrence 
greater  than  1  x  10 1in  that  life. 

Continuously  experienced. 

B 

Probable 

Will  occur  several  times  in  the  life  of  an  item,  with  a  probability  of  occurrence 
less  than  1  x  10'1  but  greater  than  1  x  10'2  in  that  life. 

Will  occur  frequently. 

C 

Occasional 

Likely  to  occur  some  time  in  the  life  of  an  item,  with  a  probability  of 
occurrence  less  than  10'2  but  greater  than  1  x  10'3  in  that  life. 

Will  occur  several  times. 

D 

Remote 

Unlikely  but  possible  to  occur  in  the  life  of  an  item,  with  a  probability  of 
occurrence  less  than  10-3  but  greater  than  1  x  10-6  in  that  life. 

Unlikely,  but  can  reasonably 
be  expected  to  occur. 

E 

Improbable 

So  unlikely,  it  can  be  assumed  occurrence  may  not  be  experienced,  with  a 
probability  of  occurrence  less  than  1  x  10'6  in  that  life. 

Unlikely  to  occur,  but 
possible. 

F 

Extremely 

Improbable 

So  improbable,  it  can  be  assumed  occurrence  is  impossible  probability  of 
occurrence  less  than  1  x  10'7  in  item  life. 

Extremely  unlikely  to  occur, 
but  not  impossible. 
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FCS  Hazard  Risk  Management  Matrix 


BAE  SYSTEMS 


Hazard 

Severity 


Catastrophic 

(I) 


Critical 

(II) 


Marginal 


Medium 


Medium 


Occasional 

(C) 


Probability  of  Occurrence 


Medium 


Medium 


Remote 

(D) 


Medium 


Medium 


Low 


Improbable 

(E) 


Medium 


Low 


Low 


Extremely 

Improbable 

(F) 


Low 


Low 


Low 


Negligible 

(IV) 


Low 


Low 


Low 


Low 


Low 


Low 


Hazard  Decision  Authority  Matrix 


Residual  Risk 


Integrating  Contractor  Risk 
Acceptance 


Government  Risk  Acceptance 


MEDIUM 


Program  Manager  and  Technical  Director 


Program  Executive  Officer  (PEO) 


LOW 


Technical  Director 


MGV  Program  Manager  (MGV-PM) 
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Bio  for  Jon  Derickson 
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Jon  S.  Derickson,  PE,  CSP,  Manager  FCS  ESOH,  BAE  Systems,  PO  Box  58123,  MD  C16, 
Santa  Clara,  CA  95052  USA,  telephone  ■  (408)  289-4797,  e-mail  - 

jon.derickson@baesystems.com. 

Mr.  Derickson  is  currently  the  Manager  for  Environment,  Safety  and  Occupational  Health 
for  the  Future  Combat  System  (FCS)  Programs  at  BAE  Systems  -  Ground  Systems  Division 
in  Santa  Clara,  CA.  He  was  previously  the  System  Safety  Group  Lead  for  Bradley  Fighting 
Vehicle  Systems. 

He  has  a  Bachelor  of  Science  in  Agricultural  Engineering  from  California  Polytechnic  State 
University  in  San  Luis  Obispo,  CA  and  a  Masters  in  Computer  Engineering  from  San  Jose 
State  University.  He  is  a  Registered  Professional  Safety  Engineer  and  a  Certified  Safety 
Professional  in  system  safety.  He  has  over  20  years  experience  in  system  safety  that 
includes  commercial  products,  military  combat  vehicles,  explosive  devices,  large  rocket 
motor  manufacturing,  and  autonomous  ground  vehicles. 
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Jon  Derickson  -  ESOH  Manager,  BAE  Systems  -  Ground  Systems  Division 


22 
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Agenda 


•  What  data  are  we  talking  about? 

•  Background 

-  Changes  in  environment 

-  Data  management  yesterday,  today,  and 
tomorrow 

•  GEIA/ANSI  STD  859  and  HDBK  859 

-  Results  to  date 

-  Related  support  to  DM  community  of  practice 

•  Example  HDBK  content 

•  Summary 


Focus  of  859  projects 


Types  of  Data 

Focus  of  859  projects 

Type 

Usage 

Examples 

Product 

Cost,  schedule  and  performance  data. 

Scientific  data  such  as  written  notes  and 

Collaboration 

observation  data. 

Engineering  drawings  and  models,  parts 
catalogues,  software  applications  and 
documentation,  operational  and  maintenance 
instructions,  and  training  materials. 

Business 

Plans  and  schedules,  financial  information 

Collaboration 

(budgets,  bases  of  estimate,  EVMS  data..,) 
inventory  status,  and  human  resource  info. 

Operational 

Transactional  Records 
Exchange 

Orders,  issues,  receipts,  bills  of  lading,  and 
invoices. 

LMI 
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Changing  Environment 


Vertical  Integration, , , ,  . . ...Trust-Based  Relationships 

DoD  Design  Bureaus. ,  .^9$P,9 P.^lkllPDf . .Industry  Design  Teams 

Military  Standards  Development  &  Implementation  Commercial 


12-15  years. 


Acquisition  Cycle  2  -  5 


years 


1950  Computer  Systems  Development  Cycle  24Mon1hs^^  ^ 


20  years  Weapon  System  Life  Cycle  5q+ 


years 


. „  Commercial  Systems  Life  Cycle  ^ 

10  -  15  years . .................... ....L..... . Years 

Logistics  Support 

DLA,  DoD  Depots  &  IMAs . . . Mix  of  Government  and  Contractor  Logistics  Support 

Transaction-based . . . . . ™ . Performance-based 


LMI 
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Transition  to  Industry  Standards 


DoD  Mandates  Contemporary  Practices 

Based  on  Sound  Principles 


MIL-STD-973 

Configuration  Mgt 

MIL-STD-2549 1 

CM  Data  Exchange 

DoD  5010.12-M 

Acq.  &  Mgt  Tech  Data 


Evolve 

Standards 


ANSI/EIA-649/A 
HDBK  61 


Broaden 
applicability: 
commercial  and 
government 


Leverage  the  latest  mgt. 
methods  and 
technologies 


Continued 

^harmonization”  with 
related  initiatives,  e.g,, 
STEP,  PDM,  IDE, ... 
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Essential  Changes:  Data 


Was 

|S  1 

Delivery  Medium 

•  Paper 

•  Electronic 

What  constitutes 
delivery 

•  I  mail  it,  you  open  it 

•  I  post  it  on  web  site,  we 
access  it 

Standardization  of 

deliverables 

•  DIDs 

•  Use  mandatory 

•  Tailoring  permitted,  but  made 
intentionally  difficult 

•  DIDs  radically  tailored  or 
ignored  entirely 

Data  environment 

•  Slow 

•  Bulky,  paper  storage 

•  Fairly  standard 

•  Limited  number  of  copies 

•  Sometimes  hard  to  find  or 
obtain  copy 

•  Rapid  to  instantaneous 

•  Compact  electronic  storage 

•  Non-standard 

•  Essentially  infinite  number  of 
copies 

•  Still  difficult  to  find 

Availability  in 
future 

•  Infinitely  available  and 
interoperable  as  long  as  copies 
not  misplaced 

Electronic  formats  subject  to 
rapid  technological 
obsolescence 

LMI 


Data  Management  Process  (Yesterday) 


Data  Mgr 


Legend: 

Government  Activities 
Contractor  Activities 


Sequential  Process 

Had  a  dedicated  and  experienced  Data  Manager 
Assumed  all  contract  required  data  is  delivered 
Philosophy  was  “Buy  Everything!” 

Only  applied  to  some  (contracted)  data 


Evolving  Data  Management  Process  (Today  and  Tomorrow) 


C  O  V if  R  N  M  ENT  CO  N  5  U  LT  t  N  Ci 


Components  of  the  New  Data  Management 


Principles 

•  Basic  tenets 
and  values 

•  General 


Practices 

•  Implementation 
specifics 

•  Some  will 
always  be 
organization- 
specific 


Professionalizing  the  Workforce 
Through  Training 


LMI 
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Purpose  and  Scope  of  Std  859 


•  Purpose 

-  Provide  a  contemporary,  principles-level,  guide  to 
requirements  for  acquisition  and  management  of  data  across 
the  product  life-cycle 

-  Enable  sharing  of  data  among  trading  partners 
...  in  a  performance-based  environment 

•  Scope 

-  Common  principles 

-  Related  enablers 

-  (Some)  key  practices 


LMI 
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DM  Principles  of  Standard  859 


Area 

Principle 

1 

Focus  and  Scope 

Define  the  organizationally  relevant  scope  of  data 
management 

2 

Customer  Support 

Plan  for,  acquire,  and  provide  data  responsive  to  customer  ; 

requirements. 

3 

Business  Context 

Develop  DM  processes  to  fit  the  context  and  business 
environment  in  which  they  will  be  performed. 

4 

Identification 

Identify  data  products  and  views  so  that  their  requirements 
and  attributes  can  be  controlled. 

5 

Change  Management 

Control  data,  repositories,  data  products,  data  views,  and 
metadata  using  approved  change  control  processes. 

6 

Data  Rights 

Establish  and  maintain  an  identification  process  for  intellectual 
property,  proprietary,  and  competition-sensitive  data. 

7 

Data  Retention 

Retain  data  commensurate  with  value  to  the  organization. 

8 

Process  Improvement 

Continuously  improve  data  management. 

9 

DM/KM  Connection 

Effectively  integrate  data  management  with  knowledge 
management. 

LMI 
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Principle  3 


Develop  Data  Management  processes  to  fit  the  context  and  business 

environment  in  which  they  will  be  performed. 
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Enabler  3.1 :  Determine  the  complete  set  of  requirements  that  the  DM 
solution  must  address 


•General  requirements 
•Data  capabilities 
•Data  processes 
•Intended  use  of  the  data 
•Related  business  objectives 
•Technology  issues 


•External  constraints 


Purpose  and  Scope  of  HDBK  859 


Provides  the  first  level  of  ‘how’ 

-  examples  and  samples 

-  approaches 

-  tools 

-  methods 

-  mini-case  studies 

-  ...to  illustrate  how  to  implement  DM  in  accordance 
with  the  principles  (compliant  with  the  standard) 


LMI 
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Example  Elements  of  Data  Strategy 


Question 

Information 

What  is  the  appropriate 
placement  model  for  the 
enterprise  DM  functions? 

Centralized — all  DM  responsibilities  are  within  one  organization 

Decentralized — DM  responsibilities  are  distributed  throughout  the 
organization 

Hybrid — one  organization  has  most  of  the  DM  responsibilities,  but 
leverages  other  enterprise  assets  (for  example  other  organizations 
provide  CM  or  records  management  resources) 

What  data  are  needed  and  by 
whom? 

Evaluate  program  life  cycle 

Identify  the  documentation  that  defines  program  requirements 

Review  documentation  and  develop  list  of  required  data  products 

Determine  who  (organizations,  functions,  or  systems)  will  use  the 
data,  what  decisions  or  functions  are  supported  by  the  data,  and  how 
and  when  they  will  use  various  types  of  data 

LMI 
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Example  Elements  of  Data  Strategy,  cont. 


Question 

Information 

Which  views  of  data  will  be 
required? 

Identify  the  content  and  format  for  each  data  product 

Identify  duplicate  data  products  or  opportunities  to  combine  data 
products 

Consolidate  requirements  for  similar  data  products 

Develop  complete  list  of  data  products  to  be  created  for  the  program 

When  will  the  data  be 
required? 

Identify  the  users  for  each  data  product  required 

Establish  whether  multiple  deliveries  are  required 

Determine  delivery  dates  and  identify  updates  or  reversions 

Prepare  a  list  of  data  products  with  users  and  delivery  dates 

What  are  the  delivery 
mechanisms? 

Evaluate  data  product  requirements 

Identify  data  product  sources  or  providers 

Identify  external  and  internal  data  providers 

Verify  that  requirements  for  data  identification  are  defined  and 
communicated 

Develop  a  list  of  data  products  with  all  sources  identified 

What  organization  design  and 
staffing  changes  are  needed? 

Determine  if  a  change  in  the  way  DM  is  staffed  requires  specific 
training  and  certification  for  data  management  personnel.3 

LMI 
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Questions  to  Help  Ensure  Completeness  of 
Requirements 


Question 

Example 

Who  determines  the  quality  of  the  data? 

Customer,  manufacturing,  engineering 

How  are  the  data  to  be  acquired? 

In-house,  contracted  for,  computer  accessed 

How  are  the  data  to  be  stored? 

File  cabinets,  servers,  microfilm 

How  will  the  data  be  protected? 

Vaults,  computing  access  control,  restricted 
buildings,  duplicate  copies  at  different  site 

Who  needs  access  to  the  data? 

Foreign  nationals,  finance,  release  group 

What  view  of  the  data  do  they  need 

Parts  catalog  for  customer,  detail  drawings  for 
manufacturing,  ad  hoc  reports  for  management 

How  are  the  data  to  be  used? 

Certification,  functional  test,  build 

How  are  the  data  to  be  delivered? 

Electronically,  paper,  on-demand  reports, 
overnight  updates  of  the  systems 

How  are  the  data  to  be  disposed  of? 

Shredding,  burning,  deletion  of  files,  garbage  can 

LMI 
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Aligning  DM  with  Contemporary  View  of  Innovation — 

“Chain  Link”  Model  of  Continuous  Improvement 


Knowledge  of  Customer  Needs 


> 

< 

Knowledge  of  Te 

i  ” 

chnical  Capability 

1 

1 

Technical  Research  &  Development 


i 


4i  LMI 


Adapted  from  S  .J.  Kline  and  N.  Rosenberg.  “ 
Rosenberg  eds.,  The  Positive  Sum  Strategy 


if  Innovation”  in  R.  Landau  and  Nj 
m:  JNational  Academic  Press,  1986. 


Summary 


•  New  environment 

-  Technology 

-  Business -trust-based,  performance-based 

•  Needed  a  fresh  way  to  think  about  DM 

-  New  recipes,  improved  methods 

•  Principles  for  the  “new”  DM  in  GEIA/ANSI  STD 
859,  methods  in  GEIA/ANSI  HDBK  859 

•  Example  HDBK  859  content 
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Data  Management  Support 
for  Modeling  and  Simulation 


Virginia  Stouffer, 
LMI 


Denise  Duncan,  LMI 
GEIA  Data  Management  Panel 
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Agenda 


•  Data  Management 

-  Changes  in  environment 

-  New  support  to  Data  Management 

•  Standard 

•  Handbook 

•  COP  and  BOK 

-  Data  management  yesterday,  today,  and 
tomorrow 

•  Modeling  and  Simulation  DM  Challenge 

•  Example  DM  solutions  and  approaches 


•  Summary 


Which  Data  Are  We  Talking  About? 


Focus  of  859  projects 


Type 

Usage 

Examples 

Product 

Cost,  schedule  and  performance  data. 

Scientific  data  such  as  written  notes  and 

Collaboration 

observation  data. 

Engineering  drawings  and  models,  parts 
catalogues,  software  applications  and 
documentation,  operational  and  maintenance 
instructions,  and  training  materials. 

Business 

Plans  and  schedules,  financial  information 

Collaboration 

(budgets,  bases  of  estimate,  EVMS  data..,) 
inventory  status,  and  human  resource  info. 

Operational 

Transactional  Records 
Exchange 

Orders,  issues,  receipts,  bills  of  lading,  and 
invoices. 

LMI 
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Acquisition  Environment  Changes 


Vertical  Integration, , , ,  . . ...Trust-Based  Relationships 

DoD  Design  Bureaus, ,  .^9$P.9 . . . . . . .Industry  Design  Teams 

Military  Standards  Development  &  Implementation  Commercial 


12-15  years. 


Acquisition  Cycle  2  -  5 


years 


1950  Computer  Systems  Development  Cycle  24Mon1hs^^  ^ 


20  years  Weapon  System  Life  Cycle  5q+ 


years 


10  -  15  years. 


Commercial  Systems  Life  Cycle 


.2  -5  Years 


DLA,  DoD  Depots  &  IMAs...,,.. . . Mix  of  Government  and  Contractor  Logistics  Support 

rp  ..  u  A  Logistics  Support  0  f  u  , 

Transaction-based . . . . ® . . KK . Performance-based 
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Impact  on  DM  Environment 


From 

To 

Medium  of  Delivery 

Paper 

Electronic 

Delivery  is 

1  mail  it,  you  open  it 

1  post  to  the  Web  portal, 
you  access 

Data  environment 

•Slow 

•Bulky,  paper  storage 

•Fairly  standard 

•Limited  number  of 
copies 

•Sometimes  hard  to  find 
or  obtain  copy 

•Rapid  -^instantaneous 

•Compact  electronic 
storage 

•Non-standard 

•Essentially  infinite  number 
of  copies 

•Still  difficult  to  find 

Future  availability 

Infinitely  available  and 
interoperable  as  long  as 
copies  not  misplaced 

Electronic  formats  subject 
to  rapid  technological 
obsolescence 

LMI 
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New  Support  to  DM:  Industry  Standards 


DoD  Mandates  Contemporary  Practices 

Based  on  Sound  Principles 


MIL-STD-973 

Configuration  Mgt 

MIL-STD-2549 1 

CM  Data  Exchange 

DoD  5010.12-M 

Acq.  &  Mgt  Tech  Data 


Evolve 

Standards 


ANSI/EIA-649/A 
HDBK  61 


Broaden 
applicability: 
commercial  and 
government 


Leverage  the  latest  mgt. 
methods  and 
technologies 


Continued 

^harmonization”  with 
related  initiatives,  e.g,, 
STEP,  PDM,  IDE, ... 


Components  of  New  Data  Management 


•  Basic  tenets 
and  values 

•  General 


Practices 

•  Implementation 
specifics 

•  Some  will 
always  be 
organization- 
specific 


Professionalizing  the  Workforce 
Through  Training 
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Purpose  and  Scope  of  HDBK  859 


Provides  the  first  level  of  ‘how’ 

-  examples  and  samples 

-  approaches 

-  tools 

-  methods 

-  mini-case  studies 

-  ...to  illustrate  how  to  implement  DM  in  accordance 
with  the  principles  (compliant  with  the  standard) 


LMI 
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Example  Content  from  HDBK  859 


aLeads  to  notional  data  archive  preservation  process 
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More  Recent  Acquisition  Changes: 


•  System  of  Systems  (SoS)  architectures: 

-  Choice  of  platform 

-  Choice  of  sensors 

-  Choice  of  munitions 

-  Choice  of  communications 

■  ■  ■ 

•  Net-Centricity 

•  Lifecycle  Logistics 
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Effects  of  Recent  Changes:  Increased  Use 
of  Modeling  and  Simulation 


•  To  evaluate  competing  architectures  before 
investing  too  much  in  to  proofs  of  concept, 
etc. 

•  To  evaluate  full  lifecycle  costs  of  competing 
concepts 

-  Much  more  modeling  earlier  in  the  lifecycle  of  the 
acquisition,  and  throughout  the  lifecycle 


LMI 
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M&S  Impacts  on  DM,  and  vice  versa 


•  Impact  on  Data  Management 

-  It  multiplies  the  amount  of  M&S  data  significantly 

-  Leads  to  M&S  being  applied  at  many  more  phases  in  the 
lifecycle 

-  Also  applied  much  earlier  -  before  DM  stood  up  in  most  cases 

•  How  can  Data  Management  help? 

-  Manage  M&S  products 

-  Manage  M&S  data 

-  Manage  ‘provenance’  of  M&S  results 


LMI 
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Example:  M&S  Before  Investing  ... 


Graduation  of  M&S  Data  Flow 


R&D 


Level  of  M&S  Data: 


TECHNOLOGY 

DEVELOPMENT 


CON 


MS  B 


MS  C 


SYSTEM 

INTEGRATION 


EPT 


SYSTEM 

DEMONSTRATION 


PRODUCTION 


OPERATION 


Trade  Studies 
Analogies 


Parametric 


Simulation 


Operational 

Processes 


Instructions  & 
Procedures 
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How  DM  Can  Help 


•  Both  DM  and  CM  have  evolved  away  from 
predictable,  clerical  functions  to  a  broader 
perspective. 

•  DM  especially  is  shifting  to  a  corporate,  strategic 
perspective  to  identify  what  types  of  data  will  be  of 
importance,  to  various  audiences,  over  the  data’s  and 
system’s  entire  lifecycle. 

-  Can  accommodate  the  need  to  capture  data  earlier  in  the 
5000.2  process  (from  JCIDS  on...?) 
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Purpose  and  Scope  of  Std  859 


•  Purpose 

-  Provide  a  contemporary,  principles-level,  guide  to 
requirements  for  acquisition  and  management  of  data  across 
the  product  life-cycle 

-  Enable  sharing  of  data  among  trading  partners 

•  Scope 

-  Common  principles 

-  Related  enablers 

-  (Some)  key  practices 


Comprehensive 

standard 


DM  Principles  of  Standard  859 


Area 

Principle 

1 

Focus  and  Scope 

Define  the  organizationally  relevant  scope  of  data 
management 

2 

Customer  Support 

Plan  for,  acquire,  and  provide  data  responsive  to  customer  ; 

requirements. 

3 

Business  Context 

Develop  DM  processes  to  fit  the  context  and  business 
environment  in  which  they  will  be  performed. 

4 

Identification 

Identify  data  products  and  views  so  that  their  requirements 
and  attributes  can  be  controlled. 

5 

Change  Management 

Control  data,  repositories,  data  products,  data  views,  and 
metadata  using  approved  change  control  processes. 

6 

Data  Rights 

Establish  and  maintain  an  identification  process  for  intellectual 
property,  proprietary,  and  competition-sensitive  data. 

7 

Data  Retention 

Retain  data  commensurate  with  value  to  the  organization. 

8 

Process  Improvement 

Continuously  improve  data  management. 

9 

DM/KM  Connection 

Effectively  integrate  data  management  with  knowledge 
management. 

Applying  DM  Principles  in  a  M&S 
Scenario 


1 .  Define  the  organizationally  relevant  scope  of  data  management 

•  Data  managers  now  think  about  the  strategic  role  of  data  management  to  the 
enterprise  vs.  a  focus  limited  to  contractual  data,  CDRLs  and  SDRLs. 

2.  Plan  for,  acquire,  and  provide  data  responsive  to  customer  requirements. 

•  A  holistic,  full  lifecycle  perspective  is  used  by  all  data  managers.  The  DM  plan 
includes  scheduled  reviews  of  the  plan  itself  to  ensure  it  remains  properly 
responsive  to  requirements. 

3.  Develop  DM  processes  to  fit  the  context  and  business  environment  in 
which  they  will  be  performed. 

•  In  a  large,  multi-tiered  M&S  environment,  perhaps  a  federated  data 
management  approach  is  best  during  the  most  dynamic  parts  of  the  lifecycle, 
for  example;  in  a  smaller  environment,  a  different  DM  architecture  could  be  a 
better  fit 

4.  Identify  data  products  and  views  so  that  their  requirements  and  attributes 
can  be  controlled. 

•  Each  version  (in  parameters,  models,  sets  of  models,  scenarios...)  uniquely 
identified  so  that  each  can  be  retrieved  and  combined  correctly;  each  result 
can  be  reproduced 


Applying  DM  Principles  in  a  M&S 
Scenario 


5.  Control  data,  repositories,  data  products,  data  views,  and  metadata  using 
approved  change  control  processes. 

•  The  data  manager  will  apply  CM  principles  to  DM  items,  tailored  to  the 
business  context.  Solutions  will  differ  based  on  various  factors;  an  example  is 
the  boundaries  with  CM  functions  in  the  participating  organization(s). 

6.  Establish  and  maintain  an  identification  process  for  intellectual  property, 
proprietary,  and  competition-sensitive  data. 

•  In  a  multi-partner  M&S  environment,  and/or  one  that  includes  proprietary 
commercial  technology,  this  can  both  be  complex  and  entail  risk  .  A  Bill  of 
Information  approach  might  be  a  good  fit  to  manage  this  complexity. 

7.  Retain  data  commensurate  with  value  to  the  organization. 

•  Data  must  be  retained  for  legal,  contractual,  and  entrepreneurial  value — but 
retention,  migration,  and  disposal  all  have  costs  associated  with  them. 

8.  Continuously  improve  data  management. 

•  Well-designed  DM  processes  include  metrics  to  monitor  the  quality  and 
efficiency  of  DM. 

9.  Effectively  integrate  data  management  with  knowledge  management. 
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Summary 


♦  There’s  a  new  appreciation  of  M&S 

-  Growth  in  number  of  M&S  efforts  and  where 
applied 

•  It’s  not  your  father’s  DM 

-  Big  changes  in  perspective,  tools,  environment 

-  Rate  of  change  depends  on 

•  exposure  to  Standard,  Handbook,  BoK,  CoP,  Training 

•  Staffing  (both  Library  and  Computer  Science 
applicable) 

*  Doing  both  well  is  worth  the  investment 
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Enabling 

Army  Level  Risk  Mitigation 


ILS.  ARMY  COMBAT  READINESS  CENTER 


EDMOND  35-01 


Fatalities 


ASP5-18 


Accidental  Fatalities 


total  30^ 
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As  of  04  October  05 
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Combat 
Vehicle  (11) 


PCC5-12B 


Is  this  an  Enemy  Threat? 


Risk  Management 


Risk  Management 


integrating 

Program 

Automated  processes 
or  od  hoc  c|uery  support 


Predictive 

Analysis 

Neurol  networks,  fuzzy  logic 
artificial  intallignnrr! 

software 


>gi.cal,  text  or  code-based 
search  engine  software 


'here's  the  Risk? 

COMPOSITE  I 


Ac  m  il f;  in 


Unit 

COPS  Strenyth 


Risk  Management 
Band  nf  Excellence 


Accident^  Caused  by 
Experience 


kXPNRIt-NCh  LhVPL 


Known 
facts  back 
to  the  Field 


Fatalix^lUata 

Combat,  fiftSjpal, 
CriminallfeliSfljicide 


Report  1 
Loss  D 

Immediate 
Notification  s 


-'The  Army 

Operations  &  Training 


Acquisition  Safety  Support 


Acquisition  Decision  Chain  Safety  Support  Chain 
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Risk  Management  Roles 

SAAE 

-  Define  Army  safety,  health,  and  environmental  risk  management 
policies  and  act  as  the  risk  decision  authority  for  high  risk  residual 
hazards  associated  with  Army  systems. 

-  Fund  and  evaluate  safety,  health,  and  environmental  research  and 
development  programs  to  address  resolution  of  generic  systemic 
safety,  health  and  environmental  problems. 

■  PEO 

-  Safety  Officer  for  assigned  systems.  Act  as  the  risk  decision 
authority  for  medium  risk  residual  safety  hazards. 

■  PM 

-  Responsible  for  identifying  all  hazards,  eliminating  or  mitigating 
when  possible,  and  providing  an  assessment  of  hazards  that  are 
not  eliminated. 

■  DASAF 

-Assist  integrating  agents,  provide  Risk  Management 
information,  assess  Risk  Management  performance 
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Reference  AR  70-1  &  AR  385-16 


System  Safety  Primary 

Objectives 

■  Ensure  hazard  control  measures  are  designed  in 
up  front  &  not  trained  out 

■  Ensure  lessons  learned  are  applied  to  new  developments; 
don’t  reinvent  the  wheel  (TIMING  DEPENDENT-  you’ve  got 
to  get  in  early  to  apply  them) 

■  Ensure  hazards  are  “risk  managed”;  residual  risk 
accepted  by  the  appropriate  authority. 

b  Apply  risk  management  throughout  the  life  cycle 
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AR  385-16 


Relative  Cost  Of  A  Change  (%) 


Implement  System  Safety  Early 


Concept  Develop/Demonstration  Production  Operation 

System  Engineering/Acquisition  Phases 
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Pay  Me  Now,  or  Pay  Me  Much  More  Later!!! 


Order  of  Precedence 


Lowest  Risk 


Highest  Risk 


Transfers  Risk 
to  the  Field 


Design  for  Minimum  Hazard 

-  Best  to  design  risk  out  of  System 
Incorporate  Safety  Devices/Features 

-  If  can’t  design  out,  design  controls  in  (H/W  Devices 
&  S/W  Features  as  Interlocks) 

Provide  Warning  Devices 

-  Generate  adequate  visual  or  audible  warning  signal 
Develop  Procedures  &  Training 

-  Susceptible  to  Personnel  Turnover 

-  Susceptible  to  Human  Error 


The  goal  is  to  "design  out"  not  "train  out"  hazards 
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IMPROVING  COMBAT  READINESS 
THROUGH  DESIGN  SELECTION 


Composite  Risk  Manasement 
Occurs  at  Each  Level 


Unit  Level 
Higher  Level 
Army  Level 

Risk  Management  occurring  at  the  system 
level  and  throughout  the  lifecycle  prevents 
unnecessary  safety  challenges  for  Soldiers 

OHIgher  Level:  Additional  procedural/training  controls 
tax  available  manpower  and  mission  effectiveness 
with  no  reduction  in  the  severity  of  the  risk 


-  Risk  transferred  to  the  Soldier  reduces  mission  effectiveness 
-Develop/modify  TTP 

-  Provide  training  range 

-  Provide  additional  manning  to  support  increased 
operational  tasks 

-  Mission  resources  diverted  to  training 

-  Increases  exposure  to  hazard 


Both  the  Risk  and  the  Controls  transferred  to  the  Soldier 
through  procedures  and  training 

-TTP 

-  Rollover  Drills 

-  Water  Egress  Drills 

-  Increases  task  load 

-  Subject  to  human  error 

-  Limited  risk  reduction 

-Does  not  reduce  severity; 
reduces  probability  by  only  one  level 


IQ  Army  Level;  Best  position  for  risk  mitigation— SOLDIERS 
CANT  AFFORD  TO  PAY  FOR  ARMY  LEVEL  HAZARDS 


-  Hazard  identified,  assessed  and  controlled  to  an 
acceptable  level  of  risk  (using  Order  of  Precedence) 

-  Possible  control  alternatives: 

-  Design:  alternate  egress  access  when  inverted 
(reduces  severity  and  probability) 

-  Safety  Devices:  combat  door  latch  wrenches 
(reduces  probability  only) 

-  Residual  risk  reduced  to  level  acceptable  at  the  PM  Level 

-  Residual  risk  mitigated;  not  transferred  to  the  Soldier 

-  Lower  order  of  precedence  controls  (i.e,  TTP)  would 
have  required  risk  acceptance  at  the  AAE  level 

-  Procedural/training;  Rollover  Drill _ 


V'  '• 


Losses  are  at  a  Unit  Lcvcl|0unitLevf  "!vel'  Soldiercan 

never  get  rrd  of  the  hazard— 

IT  WILL  ALWAYS  BE  WITH  HIM 


but  all  Controls  are  not! 
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Summary 

USACRC  supports  the  ASP  by — 

-  Reviewing  total  Army  operations  from  platoon- 
level  to  HQDA-level  daily  to  identify  RMI 
opportunities  for  keeping  soldiers  safe 

-  Providing  information  &  tools  that 
commanders  can  use  to  make  informed 
risk  decisions 

-Assessing  risk  management  performance 


Supporting  commanders' 
safety  programs  worldwide 


DASAF3-15 


BACKUP  SLIDES 


RISK  MANAGEMENT 
PROCESS  FLOW 


Where  we're  engaged  in  supporting  acquisition  safety 


Developing  hazards/ 
controls  information 
for  disseminating 
historical  safety 
lessons  learned 
for  new  systems 
(ASMIS-1) 
Synchronizing 
acquisition  & 
safety  policies 
Reviewing  DAU 
coursework  safety 
content 


a  -DSOC 
1  -DoDESOHIPT 

1  -JSSC 

I  ■  Safety  Coord.  Panel 
<  -Acquisition 
Safety  IPT 


Iniliieflieiti 

controls 

A jjj/'  "■ 


■  Safety 
Campaign  Plan 

■  JSSC  MOA  “System 
Safety  in  Materiel 
Acquisitions” 

■  DASAF  Memo 
“Eliminating 
Hazards  through 
Design  Selection” 

■  ASA(ALT)  Bulletin 


Providing  Independent  Safety  Assessments  at  MDRs/IPRs  for  ACAT I  &  II 
Participating  in  program  IPTs  &  SSWGs  to  provide  proactive  guidance 
Conducting  Accident  Investigations  of  selected  accidents 
Review  of  System  Safety  Risk  Assessments  &  Safety  Notification  Messages 


E  DIVIO  NDS5-02 


I  Ground  Bde/Bn  Cdrs  J 


U.S.  Army  Combat  Readiness  Center  (CRC) 


TF  Ground 
TF  Air 
T^Drivinc^ 


Composite  Risk  Management 
Is  The  Key  To  Success 


'aining  or  Combat, 
s  still  the  same  5-step 
ocess  and  it  works! 


1.  Identify 
Hazards 


2.  Assess 
Hazards 


3.  Develop 
Controls  & 
Make 
Decisions 


4.  Implement 
Controls 


5.  Supervise 
&  Evaluate 


Assessing  risk  from  multiple  hazards  cumulatively! 


Military/Civilian/Contractor/Terms 


G-5  Future  Ops 
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Powell 

Operational  Research 
&  Analysis 


Systems  Safety 


Programs  & 
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Futures 
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Research  Analyst 
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Safety  Engineer  GS-14 
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Safety  Mgr  (GS-018-12) 


Langhammer 


GS-018-13  Tech  Rev 


Ramsey 


Giffin 


Edmonds 


GS-803-13Tech  Rev 


Wren 


CW-4 


Berra 


GS-018-12  Tech  Rev 


GS-803-13Tech  Rev 


8  Contractors 


Kaltz 

Snyder 

Pollard 

Powell,  B 

Taylor 

V 

J 

SUPERVISE  &  EVALUATE 

"FEEDBACK" 


USACRC  provides  the  independent 
“honest  broker”  feedback 


How? 

-  System  Safety  Advocacy 

-  ISA’s 


t[j  Identify 
Hazard) 


J  Develop  ^ 
Controls  &  Make 
Decisions 


r  > 


Implement 

Controls 


If  we  don’t  perform  this  step  of  the  cycle, 
the  risk  management  process  is  incomplete. 
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DOCTRINE 

*  RM  connected  to  Strategic  Plan 
-  Embed  emerging  but  mature 
doctrine  &  TIP 
-FM 101-5 


B  RM  standardization 

•  RM  in  tng  development 

♦  RM  integration  in  lesson  plans 


COMBAT  DEVELOPMENT 

*  Human  performance  synergy 

•  Accident  investigation  frfoRowup 
-  Materiel  issues 

•  SSRA  in  Battle  Labs 


LEADER  KV*  ‘ 
DEVELOPMENT^'" 

•  Embed  RM  in  BCTP 

•  Embed  RM  in  Prairie  Warrior 
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OPPORTUNITY  FOR 
WORLD  CLASS  SAFETY 
PERFORMANCE 
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THE  ROR&E 
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ROME 


Integrating  Agnnt:  CG  AMC 

- - 

Materiel  acquisition  policy  chang 

•  Strengthened  acquisition  proce 
by  codifying  system  safely  risk 
assessment  procedures 

•  World  class  performance 
by  PEOs  and  PMs 

•  Handoff  info  about 
hazards  to  soldiers 


A  Major  Subjective  Analysis 


AH  “Line-Item  Inventory”  Hazard 
Analysis  /  Risk  Assessment 
methods  *  suffer  this  shortfall: 


THE  ANALYTICAL  CONSTRUCT 
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•  Examples: 

Preliminary  Hazard  Analysis 
Failure  Modes  and  Effects  Analysis 
Functional  Hazard  Analysis 


Shortfall. 


THE  ANALYST’S  VIEW 
of 

SYSTEM  RISK 


RISK  SUMMATION 
METHODS  ARE 
NEEDED! 


REAL  WORLD 
SYSTEM  RISK 


■\ 


2  (s1  x  p,)  + 
(s2  x  p2)  + 

(s3  x  p3)  + 
(s„  x  P„)  + 
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Establishing  Safety  Performance  in 
the  Capability  Identification  Process 


PROBLEM: 'Historical  safety  lessons 
learned,  accident  data,  known 
hazards,  etc.  are  not  leveraged 
In  the  selection  nrocess. 
Preliminary  hazard  analyses 
do  not  occur. 


Strategic  Policy  Guidance 


Joint  Operations  Concepts 


Joint  Operations  Concepts 
Joint  Functional  Concepts 
Integrated  Architectures 


SUPPORT:  System  safety  resources  needed: 

system  safety  engineers  in 
all  combat  developments 
*  hazards  information  from 
accident  data 


DOTMLPF  Change 
Recommendation 


“Safely  should  be 
a  requirement  up 
froni  and  across 
(he  DOTMLPF” 

-  Hon,  Claude  Bolton,  ME 


Functional  Solution  Analysis 


RESULTS:  Safety  performance 
criteria  established  in 
requirements  documentation. 


Systems  Engineering  Process 


P 

R 

O 

C 

E 

S 

S 

I 

N 

P 

U 

T 


Requirements 

Analysis 


Requirements 
Loop 


Verification 


Design 
Loop 


Loop 


System 

Safety 

applied  here. 
How  do  our 
efforts  affect 


the  design? 


♦ 


PROCESS  OUTPUT 


Where  we  can  help  you... 

■  Supporting  risk  management  decisions 

-  System  Safety  Risk  Assessments  (SSRA) 

-  Army  Safety  Action  Team  (ASAT) 

■  Providing  hazards  information  from  Army  accidents 
to  influence  design  selection 

a  Coordinating  safety  investment  strategies 
to  fund  safety  improvements 

-  Safety  Coordinating  Panel  (SCP) 

■  Analyzing  and  communicating  safety  information 

-  Countermeasure,  Flightfax,  Impax,  PLRs 
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Where  we  need  your  help... 

Ensuring  an  effective  SSMP  is  developed  &  executed 
as  part  of  the  acquisition  strategy 

■  Providing  design  solutions  for  recurrent  hazards  that 
produce  accidents 

■  Enabling  acquisition  leaders  to  routinely  assess  safety 
performance 

■  Integrating  system  safety  within  the  overall  systems 
engineering  process 

■  Establishing  safety  performance  capabilities  for  the  user 
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Where’s  the  Risk? 


RISK  MANAGEMENT 


Risk  Management 


Small  Arms 


RPG 


Successful 

Risk 

Management 


Risk  Management 
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What's  going  to  kill  me  &  my  buddies, 
Enemy  or  Accident? 
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INNOVATIVE  PROCUREMENT 

STRATEGIES 

BY 

DAVID  M.  EIBAND 


DEFENSE  ACQUISITION  UNIVERSITY,  SOUTH  CAMPUS 
6767  OLD  MADISON  PIKE,  BUILDING  7 
HUNTSVILLE,  AL  35806 


OVERVIEW 


FIRST  PROCUREMENT  FOR  PRINTED  WIRING 
BOARDS  (PWB) 

-  $2  MILLION  CEILING 

-  2  YEAR  TERM 

SECOND  PROCUREMENT  FOR  ENTIRE  CLASSES 
OF  ITEMS  FROM  COMPONENTS  TO  COMPLEX 
ASSEMBLIES 

-  $45  MILLION  CEILING 

-  5  YEAR  TERM 


FIRST  PROCUREMENT  APPROACH 


PWBs  HISTORICALLY  PROCURRED  VIA  CREDIT 
CARD  AS  MICRO-PURCHASES 


-  MULTIPLE  BUYS  <  $2,500 


-  PERCEIVED  AS  “SPLITTING” 

•  PENALTIES  ARE  $10,000  FINE  AND/OR  5  YEAR 
IMPRISONMENT 


FIRST  APPROACH  (CONT) 


SOLUTION 

-  USE  FEDERAL  ACQUISTION  REGULATION  (FAR) 
TEST  PROGRAM  AUTHORITY 

•  RAISED  SIMPLIFIED  ACQUISITION 
PROCEDURES  (SAP)  LIMIT  FROM  $100,000  TO 
$5  MILLION 

-  PROCURE  PWBs  AS  A  SUPPLY  RATHER  THAN  A 
SERVICE 

•  REQUIRED  TO  MEET  SAP  DICTATES 


FIRST  APPROACH  (CONT) 


USE  TWO  YEAR  CONTRACT 

•  LOCK-IN  PRICES 

•  DECREASE  ADMINISTRATIVE  COSTS  AND 
PROCESSING  TIME  DELAYS 

WARRANT  ORDERING  OFFICER  IN  TECHNICAL 
OFFICE 

•  AVOIDED  USING  CONTRACTING  SHOP  FOR 
EACH  BUY 


FIRST  APPROACH  (CONT) 


USE  PURCHASE  CARD  FOR  PAYMENT 

•  CONTRACTOR  PAID  IN  TWO  DAYS  WITHOUT 
DFAS  INVOLVEMENT 

•  NEGOTIATED  BETTER  OVERALL  PRICES 


FIRST  APPROACH  (CONT) 


RESULTS 

-  AWARDED  A  $1 .2  MILLION  PROCUREMENT  IN  70 
DAYS  RATHER  THAN  260  DAYS  USING  NORMAL 
CONTRACTING  BY  NEGOTIATION  PROCEDURES 

-  NEGOTIATED  LOWER  PRICES 

-  SIGNIFICANTLY  IMPROVED  PROCUREMENT 
TURN  AROUND  TIME 

-  BROUGHT  ORDERING  CAPABILITY  INTO 
TECHNICAL  OFFICE 


SECOND  PROCUREMENT  APPROACH 


PURCHASED  MYRIAD  ITEMS  RANGING  FROM 
ELECTRONIC  COMPONENTS  TO  COMPLEX 
SYSTEMS 

-  $5  MILLION  PER  YEAR 
-TIME  CONSUMING 

-  PROCUREMENTS  OFTEN  COMPLICATED 


-  OFTEN  LESS  THAN  TIMELY 


SECOND  APPROACH  (CONT) 


PROCUREMENT  STRATEGY 

-  BUY  EVERYTHING,  EXCEPT  COMPUTERS,  FROM 
A  SINGLE  VENDOR 

-  STRUCTURE  A  FIVE  YEAR  CONTRACT 

-  RETAIN  BASIC  FEATURES  OF  THE  FIRST 
PROCUREMENT  APPROACH 


SECOND  APPROACH  (CONT) 


PROBLEMS 


-  APPEARANCE  OF  “BUNDLING”  AND 
RESTRICTING  SMALL  BUSINESS 


-  FORCED  TO  INCLUDE  ANOTHER  ORGANIZATION 
RAISING  CONTRACT  TO  $45  MILLION 


•  UNABLE  TO  USE  SAP 


•  FAR  REQUIRED  CONTRACTING  BY 
NEGOTIATION  PROCEDURES 


SECOND  APPROACH  (CONT) 


SOLUTION 


-  USED  FAR  SOURCES  SOUGHT  REQUEST  TO 
ADDRESS  BUNDLING 


•  AVAILABLE  ONLY  TO  R&D  PROGRAMS 


•  EVALUATED  113  SUBMITTALS-ONLY  TWO 
VENDORS  WERE  “QUALIFIED” 


•  NEITHER  MET  THE  REQUIREMENT  TO 
PROVIDE  100%  SMALL  BUSINESS  MATERIALS 


SECOND  APPROACH  (CONT) 


NEGOTIATED  APPROACH  WITH  SMALL 
BUSINESS  ADMINISTRATION  (SBA) 

•  5%  PRICE  ADVANTAGE  TO  SMALL  BUSINESS 
SUPPLIER 


•  TWO  VENDORS,  ONE  PURELY  SMALL 
BUSINESS  AND  ONE  UNRESTRICTED 
BUSINESS 


•  COMPETE  OFFERS  USING  SBA  AGREEMENT 
TO  SELECT  BUYS 


SECOND  APPROACH  (CONT) 


WARRANTED  ORDERING  OFFICER  IN 
TECHNICAL  OFFICE  FOR  PROCUREMENTS 

UTILIZED  PURCHASE  CARD  FOR  PAYMENT 


SECOND  APPROACH  (CONT) 


RESULTS 


-  90  DAYS  TO  ACCOMPLISH  SOURCES  SOUGHT 
AND  SBA  NEGOTIATIONS 


-  APPROACH  HIGHLY  POLITICAL 

•  PROCUREMENT  EXPENDED  17  MONTHS 
FROM  START  TO  SOURCE  SELECTION 
COMPLETION 


CONCLUSIONS 


FAR  CAN  BE  USED  TO  MAXIMIZE  PROGRAM 
SUCCESS 

COMPETENT,  OPEN-MINDED  TEAMS  OPTIMIZE 
RESULTS 

-  TECHNICAL  OFFICE 

-  PROCUREMENT  AND  SMALL  BUSINESS  OFFICE 


Innovative  Procurement  Strategies 


By 


David  M.  Eiband 


Defense  Acquisition  University,  South  Campus 
6767  Old  Madison  Pike,  Building  7 
Huntsville,  AL  35806 


Abstract 


Given  downsizing  and  shrinking  budgets  in  the  Department  of  Defense  (DoD), 
most  organizations  are  searching  for  means  to  affect  savings,  operate  more  efficiently, 
and  produce  a  higher  quality  product.  At  times  those  admirable  goals  can  go  astray, 
leading  to  potentially  improper  procurements  while  striving  to  meet  mission 
requirements.  We  executed  two  innovative  procurement  experiments  that  accomplished 
those  goals  specifically  in  the  research,  development,  test  and  evaluation  (RDT&E) 
environment.  R&D,  by  its  nature,  does  not  deal  in  commonly  available  systems,  and  our 
first  experiment  pushed  the  procurement  envelope  by  repetitively  procuring  unique 
developmental  hardware  through  use  of  the  Simplified  Acquisition  Procedures  defined  in 
the  Federal  Acquisition  Regulation  (FAR)  combined  with  aggressive  use  of  the 
Government  purchase  card  as  a  payment  vehicle.  This  approach  eliminated  the  very 
common  risk  of  “splitting”  procurements  to  meet  purchase  card  dollar  limits  and  utilized 
Ordering  Officer  capability  resident  in  a  group  outside  of  the  procurement  organization  to 
minimize  administrative  overhead  time  and  organizational  conflicts.  The  follow-on 
experiment  expanded  upon  the  simplified  acquisition  approach  while  additionally 
aggregating  large  numbers  of  different  developmental  hardware  purchases  into  a  single, 
multi-year  contract.  The  expanded  approach  developed  a  methodology  that  avoided 
“bundling”  procurements  and  required  extensive  negotiation  with  the  Small  Business 
Administration  to  complete  the  procurement.  This  paper  will  discuss  the  evolution  of 
both  strategies,  their  regulatory  basis,  and  the  net  effect  of  the  total  experiment. 


The  Setting 


Our  group  designed,  validated,  fabricated,  tested,  and  collected  data  from 
telemetry  systems,  in  particular  weapons  applications.  Comprised  of  a  mixed  group  of 
engineers,  technicians,  and  support  specialists,  the  group  had  designed  and  built  telemetry 
systems  for  weapons  ranging  from  2.  75  inch  rockets,  to  10  inch  diameter  hypersonic 
anti-radar  missiles,  to  five  inch  diameter  air-to-air  missiles,  to  one  inch  bomb  fuzes,  to 
foreign  military  system  evaluation.  In  all  applications,  our  designs  required  significant 
procurement  effort  to  fabricate  and  test  systems,  and  when  appropriate,  our  designs  were 
also  carried  internally  into  prototype  production.  Professional  technical  image  aside,  our 
work  could  not  be  completed  without  continuing  forays  into  the  world  of  Government 
procurement. 


Procurement  Background 

Within  the  Government  and  certainly  within  the  DoD  RDT  &E  environment, 
procurements  follow  a  common  preferred  evolution.  When  possible,  procurements  are 
first  made  using  micro-purchase  procedures,  which  are  less  than  or  equal  to  $2,500;  then 
small  purchase  procedures  which  are  greater  than  $2,500  and  no  more  than  $100,000;  and 
finally  contract  procedures  for  all  procurements  greater  than  $  1 00,000.  Procurement 
complexity,  lead  time,  and  administrative  costs  all  increase  as  one  ascends  the  hierarchy 
from  group  to  group,  and  within  the  small  purchase  and  contract  ranges,  complexity,  lead 


time,  and  administrative  costs  increase  at  prescribed  cost  threshold  levels.  Clearly,  there 
are  advantages  to  keeping  individual  procurement  costs  as  low  as  possible  to  avoid  the 
noted  procurement  overhead,  but  as  discussed  later,  there  are  disadvantages  as  well. 

The  First  Experiment 

Organizations  do  not  change  their  operating  philosophy  without  reason,  and  our 
group  procurement  strategy  fits  that  mold.  For  years,  our  group  had  used  multiple 
$2,500  per  purchase  micro-purchases  to  procure  printed  wiring  boards  (PWB)  as  well  as 
a  myriad  of  other  classes  of  items.  The  PWBs,  while  very  complex  in  this  application, 
are  essentially  the  easily  recognized  nonconducting  fiberglass  boards  overlaid  with 
conducting  traces  found  in  most  electronic  devices.  Various  electronic  components, 
connectors,  and  wiring  harnesses  are  attached  to  the  traces  to  implement  a  particular 
design.  From  a  technical  standpoint,  each  PWB  is  unique  to  a  peculiar  system  with 
multiple  PWBs  used  in  each  system;  however,  the  procurement  system  does  not  treat 
PWBs  with  such  fidelity.  From  a  procurement  standpoint,  a  PWB  is  a  PWB  just  like 
every  other  PWB.  That  perceived  similarity  was  the  genesis  of  our  procurement  change. 

Using  that  logic,  the  procurement  community  believed  that  the  micro-  purchase 
agent  was  breaking  down,  more  commonly  "splitting,”  similar  PWB  procurements  to 
keep  the  individual  procurements  under  the  $2,500  threshold.  By  FAR  Part 
13.003(c)(l)(2)definition,  splitting  breaks  a  procurement  into  "several  purchases  that  are 
less  than  the  applicable  threshold  merely  to  (1)  permit  use  of  simplified  acquisition 


procedures  or  (2)  to  avoid  any  requirement  that  applies  to  purchases  exceeding  the  micro¬ 
purchase  threshold.”  Our  perceived  splitting  was,  therefore,  potentially  a  violation  of  the 
FAR.  Even  more  critically,  splitting  can  also  have  significant  legal  penalties  under  18 
U.S.C.  287  of  a  $10,000  fine  and/or  5  year  imprisonment.  In  other  words,  our  multiple 
buys  via  credit  card  were  likely  prohibited  and  corrective  action  was  required  to  eliminate 
any  potential  violation. 

With  that  newly  found  impetus,  we  embarked  to  change  our  PWB  procurement 
strategy  and  began  investigating  alternatives  to  the  commonly  utilized  micro-purchase 
approach.  Because  of  their  intentionally  structured  abbreviated  requirements,  FAR  Part 
13  Simplified  Acquisition  Procedures  for  commercial  items  apparently  offered  the  best 
option.  This  conclusion  seemed  obvious  as  the  simplified  acquisition  procedures  were 
designed  to  "(a)  reduce  administrative  costs;  (b)  improve  opportunities  for  small,  small 
disadvantaged,  and  women-owned  small. ...  (c)  promote  efficiency  and  economy  in 
contracting,  and  (d)  avoid  unnecessary  burdens  for  agencies  and  contractors.”  Of  course 
in  the  R&D  arena,  many  items  were  not  traditionally  considered  commercial,  and  the 
$100,000  limit  on  simplified  acquisition  procedure  use  loomed  as  a  significant  drawback 
as  our  annual  procurements  easily  exceeded  that  limit  by  an  order  of  magnitude. 

Fortunately,  several  factors  improved  our  situation.  FAR  13.500(a)  authorizes 
test  programs  greater  than  $100,000  but  not  exceeding  $5,000,000  for  simplified 
acquisitions,  if  the  proper  approval  could  be  secured.  Additionally  from  an 
administrative  overhead  perspective,  simplified  acquisition  offered  no-cost  local 


procurement  department  processing,  an  obviously  attractive  characteristic  not  necessarily 
available  at  all  procurement  organizations. 

The  major  inhibitor  to  that  approach  was  the  FAR  13.500(c)  test  program 
requirement  to  purchase  commercial  items,  or  supplies  in  laymen  terms,  rather  than 
services.  Historically,  the  local  procurement  office  considered  PWBs  a  service  rather  than 
a  supply  because  labor  was  required  to  manufacture  each  PWB.  A  closer  reading  of 
FAR  2.101  indicated  that  a  commercial  item  is,  by  definition,  “(  a)  any  item,  other  than 
real  property,  that  is  of  a  type  customarily  used  for  nongovernmental  purposes  and  that 
(1)  has  been  sold,  leased,  or  licensed  to  the  general  public;  or,  (2)  has  been  offered  for 
sale,  lease,  or  license  to  the  general  public.”  Needless  to  say,  PWBs  are  ubiquitous  in 
modem  society  and  clearly  met  the  FAR  definition  of  a  commercial  item.  The 
contracting  officer  conceded  that  point,  and  with  that  agreement,  the  fundamental 
procurement  approach  was  established. 

The  opportunities  garnered  by  this  strategy  greatly  benefited  the  PWB 
procurement.  By  using  a  two-year  contract,  the  administrative  costs  and  lead-time  were 
significantly  decreased  in  comparison  to  annual  reprocurements.  The  commercial  item  or 
catalog  pricing  aspects  of  simplified  acquisition  allowed  the  contract  to  be  limited  to  one 
page  of  text  for  the  Statement  of  Work  (SOW)  by  using  the  entire  matrix  of  PWB 
combinations  from  the  catalog  instead  of  individual  specifications.  Because  the  contract 
allowed  the  development  of  Government-contractor  relationships  over  a  two-year  period 


of  performance,  the  strategy  significantly  enhanced  the  potential  for  improvement  and 
efficiency  over  the  normal,  mandatorily-competed  individual  procurement. 

Commercial  item  or  catalog  pricing  also  facilitated  ordering  via  electronic  media 
and  more  importantly  allowed  ordering  using  the  Government  purchase  card  and  a 
warranted  Ordering  Officer  within  our  own  group.  Use  of  the  Government-wide 
commercial  purchase  card  and  Ordering  Officer  was  critical  because  orders  may  be 
placed  without  resorting  to  a  procurement  contracting  officer  or,  after  delivery,  to  utilize 
the  Defense  Finance  and  Accounting  System  (DFAS)  in  compliance  with  FAR  13.301(a). 
This  advantage  resulted  in  faster  turnaround  times  and  greatly  improved  contractor 
payment,  two  notably  inefficient  aspects  of  Government  procurement,  and  even 
recommended  in  FAR  13.500. 

In  the  end,  this  contract  experiment  produced  several  startling  results.  A 
$1.2  million  procurement  was  awarded  in  70  days  rather  than  the  nominal  260  days  using 
normal  FAR  Part  15  Contracting  by  Negotiation  contracting  procedures.  The  two-year 
contract  featured  fixed,  published  prices  over  the  duration  of  the  contract,  and  from  a 
technical  standpoint,  the  single  order  limit  was  raised  from  the  $2,500  micro-purchase 
limit  to  $50,000  of  PWBs  per  month.  The  contractor  benefited  from  more  efficient 
production  runs  that  are  not  constrained  by  the  micro-purchase  size  limit,  and  from  a 
business  aspect,  the  contractor  receives  payment  within  two  days  of  acceptance  rather 
than  months  or  later  after  submittal  to  DFAS.  For  these  reasons,  the  simplified 
acquisition  procedure  approach  was  clearly  a  boon  for  both  the  Government  and  industry. 


The  Second  Experiment 


Given  the  preceding  discussion,  the  first  experiment  was  obviously  limited  in 
scope  to  a  single  item  type,  and  while  the  simplified  acquisition  approach  and 
Government  purchase  card  payment  method  offered  savings,  the  administrative  overhead 
savings  applied  only  to  a  single  item  type  procurement.  The  second  experiment  was 
therefore  designed  to  significantly  expand  the  coverage  of  the  contract  while  retaining  all 
the  positive  aspects  identified  and  proven  by  the  first  experiment. 

Ninety  percent  of  all  procurements  within  our  organization  consisted  of  electronic 
components  and  devices,  clearly  making  that  area  our  new  target.  Our  procurement 
requirement  for  the  second  experiment  was  structured  to  include  classes  of  items  ranging 
from  simple  components  such  as  resistors,  diodes,  and  capacitors  to  complete  complex 
assemblies  such  as  transmitters,  global  positioning  systems  (GPS),  inertia  navigation 
systems  (INS),  and  electronic  test  equipment.  Our  historical  procurement  of  these  items 
was  over  $5,000,000  annually  and  rose  to  $9,000,000  annually  when  a  sister  organization 
was  added. 

The  magnitude  of  this  effort  had  two  immediate  impacts.  First,  since  the 
technical  requirements  would  likely  exceed  the  capability  of  any  small  or  disadvantaged 
business  to  meet  the  requirement,  the  procurement  strategy  directly  conflicted  with  the 
Government's  policy  in  FAR  19.201  to  provide  maximum  practicable  opportunities  in  its 
acquisitions  to  those  same  groups.  In  addition,  our  intention  to  maximize  the  benefits  of 
multi-year  contracting  and  therefore  structure  a  five-year,  maximum-length  contract 


further  complicated  this  strategy,  and  the  resulting  $45,000,000  contract  could  not  escape 
the  intense  scrutiny  of  the  small  business  community  or  their  champions  in  the  Small 
Business  Administration  (SBA). 


Because  simplified  acquisition  procedures  apply  to  commercial  item  contracts 
worth  no  more  than  $5,000,000  using  the  FAR-authorized  test  program,  our  planned  five 
year,  $45,000,000  approach  created  the  second  problem,  a  reliance  on  the  more 
traditional  combination  of  FAR  Part  12  Acquisition  of  Commercial  Items  and  Part  15 
Contracting  by  Negotiation  procedures.  Due  to  the  nature  of  the  procurement,  this  latter 
problem  did  not  adversely  effect  the  overall  experiment  and  could  not  be  avoided  in  any 
event;  the  first  problem  with  the  small  business  community,  however,  was  significant  and 
required  serious  attention. 

The  shear  magnitude  of  the  procurement  guaranteed  small  business  scrutiny  and 
the  consequent  potential  for  protest;  in  addition,  the  nature  of  the  required  items 
themselves  could  be  grounds  for  protest.  Because  the  procurement  covered  a  broad 
spectrum  of  items  from  electronic  components  to  complete  electronic  assemblies, 
"bundling"  became  an  issue.  By  FAR  2.101  definition,  bundling  is  the  "consolidation  of 
two  or  more  requirements  for  supplies  or  services,  previously  provided  or  performed 
under  separate  smaller  contracts,  into  a  solicitation  for  a  single  contract  that  is  likely  to  be 
unsuitable  for  award  to  a  small  business  concern  due  to:  (i)  the  diversity,  size,  or 
specialized  nature  of  the  elements  of  the  performance  specified;  (ii)  the  aggregate  dollar 
value  of  the  anticipated  award;  (iii)  the  geographic  dispersion  of  the  contract  performance 


sites;  or  (iv)  any  combination  of  the  factors  described  in  (l)(i),  (ii),  and  (iii).”  Without 
doubt,  our  intended  procurement  had  the  look  and  feel  of  bundling  and  thus  was  highly 
suspect  as  a  viable  procurement  strategy.  However  given  our  intent  to  truly  explore  the 
extremes  of  the  procurement  envelope,  we  elected  to  pursue  a  mitigating  tactic  that  would 
hopefully  preserve  our  intended  strategy  while  satisfying  the  dictates  of  the  small 
business  regulations. 

In  order  to  address  these  small  business  issues,  we  coordinated  with  our  base 
Small  Business  Office  to  release  a  "sources  sought"  request  to  industry.  The  sources 
sought  request  stated  our  requirement  and  directed  interested  contractors  to  provide  key 
data  describing  their  capability  to  meet  the  published  requirement  in  accordance  with 
FAR  5.205(a).  Available  only  to  R&D  programs,  the  sources  sought  option  perfectly  fit 
our  needs.  One  hundred  thirteen  contractors  responded  with  data,  and  the  packages  were 
evaluated  against  the  known  requirement.  Only  two  contractors  were  rated  "qualified"  as 
meeting  the  technical  requirement,  but  none  of  the  113  small  business  concerns  met  the 
FAR  19.502.2(c)  requirement  that  small  business  suppliers  must  furnish  products  from 
100%  small  business  concerns. 

Armed  with  that  information,  we  negotiated  an  acceptable  small  business  set  aside 
agreement  with  the  SBA.  The  agreement  would  allow  the  selected  small  business  to  bid, 
within  the  noted  FAR  constraints,  for  the  maximum  number  of  items  for  which  they 
qualified,  and  the  remainder  of  the  procurement  would  be  awarded  unrestrictedly  to  any 
qualified  vendor,  which  could  include  either  small  and  large  businesses.  In  essence,  the 


original  procurement  would  consist  of  two  separate  awards,  and  as  negotiated,  the  small 
business  contractor  would  be  allowed  a  5%  cost  advantage  over  the  large  business 
contractor.  However  because  the  requirement  was  identical  whether  met  by  a  small 
business  or  unrestricted  vendor,  the  contract  awards  would  vary  only  by  the  scope  of 
materials  provided  and  the  peculiar  small  business  clauses  unique  to  a  small  business 
contract. 

In  the  end,  the  second  experiment  performed  very  much  as  the  original  had.  The 
final  contract  allowed  product  ordering  without  using  the  procurement  department  or  a 
procurement  contracting  officer;  the  Government  purchase  card  system  paid  the 
contractor  after  the  acceptance  of  each  order;  and,  most  importantly,  the  contract  covered 
the  universe  of  electronic  requirements  over  a  five  year  contract  period.  When  one 
considers  that  our  group  averaged  over  750  procurement  actions  annually,  this  single 
contract  was  a  manifold  improvement  in  acquisition  overhead  costs  and  time  delays. 

Processing  the  contract  did  not,  however,  proceed  as  quickly  as  had  the  original 
procurement.  First,  the  sources  sought  and  SBA  approval  effort  required  90  days  to 
accomplish,  but  that  delay  can  be  fully  avoided  on  future  procurements  as  the  answer  is 
generically  appropriate  to  any  other  like  procurement.  Second,  no  $45  million  program 
will  quickly  advance  through  either  the  procurement  or  management  approval  process, 
and  it  did  not.  In  fact,  the  entire  process  required  over  17  months  from  initiation  to 


source  selection. 


Conclusions 


This  paper  reports  the  results  of  two  experiments  in  Government  procurement. 
The  experiments  were  evolutionary  in  nature  with  the  second  procurement  growing  upon 
the  successful  results  of  the  first.  As  was  noted,  the  procurements  tested  the  ability  of  the 
Federal  acquisition  regulations  to  optimally  meet  the  requirements  of  the  RDT  &E 
environment,  the  patience  of  the  procurement  system  in  addressing  and  eventually 
accepting  innovative  concepts,  and  the  professionalism  of  the  disparate  technical  and 
procurement  communities  in  accomplishing  a  very  different  but  mutually  beneficial  goal. 
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Motivation 


*  Over  the  next  several  years,  DoD  will  begin  fielding 
components  of  the  Net-Centric  Enterprise  Services  (NCES) 
and  Global  Information  Grid  (GIG) 

-  Testing  these  Service  Oriented  Architecture  (SOA)-based 
capabilities  will  require  new  techniques  and  tools  beyond 
those  used  for  traditional  platform-based  systems 

•  Hypothesis:  an  approach  to  testing  net-centric  systems  can 
be  formed  based  on  successful  experiences  testing  service 
oriented  distributed  simulation  systems. 

-  This  briefing  presents  the  findings  of  a  MITRE  IR&D 
study  that  examined  the  potential  for  simulation  test 
methods  and  tools  to  address  net-centric  test  challenges 


Trends  in  Net-Centric  Computing: 
The  GIG  and  NCES 
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Global  Information  Grid -Bandwidth  Expansion  (GIG-BE) 
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Enterprise  Service  Management 


Trends  in  Net-Centric  Computing: 
Communities  of  Interest  (COIs)  and  the  Blue 
Force  Tracking  (BFT)  COI 


BFT  Service  =  BFT  Data  Model  +  BFT  Sources  +  DDS  +  CES  +  Consumers 


User  Scope ... 
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BFT  COI  Architecture  (from  Blue  Force  Tracking  (BFT)  Community  of  Interest  ( COI)  Service 
(vl.O);  F.  Wildes,  K.  Kelley,  and  P.  Kim;  MITRE  Working  Note:  WN  05W0000001,  Dec.  2004.) 
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Challenges  Associated  with  Testing 
SOAs:  Why  is  it  hard? 


Rapidly  Evolving  Standards 

-  limits  potential  choreography  between  services 
Rapidly  Evolving  Core  Services 

-  many  still  in  early  prototyping  phase 
Organization  of  Registries 

-  the  right  service  is  out  there  somewhere,  can 
you  find  it? 

Service  Pedigree 

-  once  you  find  a  service,  can  you  trust  it? 


Multiple  Levels  of  Testing  will  be  Required 
for  NCES  and  the  GIG 


*  Testing  of  each  component 

-  Does  this  node  (database,  consumer  console,  service 
provider)  perform  its  function  properly  (as  expected  and 
according  to  specifications)? 

*  Testing  services  and  transport  components  working 
together  as  different  subnets 

-  Do  this/these  services  work  in  an  integrated  fashion  on 
the  network  as  expected? 

*  Testing  each  system’s  use  of  the  network 

-  Does  this  network  architecture  have  bottlenecks  and 
what  is  the  maximum  volume  it  can  handle? 

-  What  is  the  network  performance? 

*  Testing  the  end-to-end  suite  of  systems  over  the  network 


Services  Used  in  Distributed  Simulation 
The  High  Level  Architecture 


•  Service  Oriented  Architecture  for  exchanging  data  among  federated  applications 
•  Simulations,  real-world  systems  and/oror  system  emulators,  support  utilities... 


•  Calls  for  an  RTI  which  brokers  data 
exchange  via  7  service  families 

•  Includes  publication  & 
subscription  services 

•  Offers  simulation  time  clock 
services 


Runtime  Infrastructure  (RTI) 

Federation  Management 

Ownership  Management  j 

Object  Management 

Data  Distribution  Management  j 

Time  Management 
Declaration  Management 

Support  Services  j 
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Parallels  Between  the  Two  Computing 
Worlds 


Distributed  Simulation  World 


Test  Tools: 

•SITH,  RTI  Verifier,  ... 


Test  Tools: 

•  SO  A  Test™,  TestMaker™ 
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Parallels  Between  the  Two  Computing 
Worlds 


•  Lessons  learned  from  testing  distributed  simulation  systems, 
including  High  Level  Architecture  (HLA)  systems,  have  the  potential 
to  be  leveraged  to  test  net-centric  systems 

-  Both  HLA  and  NCES  are  based  on  service  oriented  principles 
that  have,  at  their  core,  a  set  of  common  infrastructure  services 
that  provide  the  basic  connection  mechanisms  necessary  for 
interoperability 

-  Both  worlds  also  embrace  the  concept  that  some  subsets  of  the 
systems  need  to  be  tightly  coupled  together  because  their 
missions  are  strongly  related  or  they  share  certain  data 
exchange  requirements 

-  Both  worlds  also  embrace  a  larger  enterprise  view  whereby 
multiple  nodes  and  multiple  system  subsets  can  interoperate  as 
needed  on  a  loosely  coupled  basis. 
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There  are  striking  similarities  between  HLA 
Services  and  Core  Enterprise  Services 


HLA  Services 
O  NCES 
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Operational  Differences  between  HLA 
and  GIG  Computing 


•  Routing  of  information  exchanges 

-  HLA:  all  data  must  pass  through  middleware  (runtime 
infrastructure) 

-  GIG:  data  routed  “directly”  between  services 

•  Most  appropriate  route  between  any  two  services  likely  to 
change  over  time 

•  Persistence  of  participants 

-  HLA:  static  set  of  federates  and  data  exchanges 

•  Addition  or  deletion  of  services  during  execution  is  not  the 
norm 

-  GIG:  can  be  open-ended 

•  Dynamic  addition  or  deletion  of  services  is  expected  during 
normal  operations 
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Testing  Expertise  in  the  Distributed 
Simulation  Community 


The  simulation  community  has  years  of  successful  experience 
testing  complex  distributed  simulation  systems,  including  High 
Level  Architecture  (HLA)  systems: 

-  Testing  individual  system  functionality  and  performance. 

-  Testing  runtime  infrastructure  services  for  correct  functionality 
and  for  performance. 

-  Testing  the  simulations  for  their  ability  to  use  the  runtime 
infrastructure  services  and  to  publish  and  subscribe  to  data  as 
specified. 

-  Testing  subsets  of  the  simulations  working  together  over  the 
RTI  services. 

-  Testing  the  end  to  end  federation  for  functionality  and 
performance. 


Distributed  Simulation  Test  Tools: 
RTI  Verifier 


•  Created  to  certify  compliance  of  RTIs  with  HLA  Interface 
Spec 

•  RTI  Verifier  consists  of: 

-  Database  of  required  tests 

-  Launcher  that  starts  federates  and  RTI 

-  Test  Controller  that  stimulates  interplay  between 
federates  and  RTI 

•  Key  component:  Script  Definition  Language 

-  For  specifying  tests 


Distributed  Simulation  Test  Tools: 
RTI  Verifier  (cont’d) 


TestFederate  TestFederate  *  ■  *  TestFederate 


A  i 

t _ * 

k- 

i 

RTI 

w 

comm  li1.!  icais  irrifj 

J 


RTI  Verifier  Architecture  (Ref.  Verifier3  User’s  Manual,  HLA  RTI  Verifier  Team,  MITRE 
Corporation,  February  2005.) 
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Distributed  Simulation  Test  Tools: 


RTI  Verifier 


RTI  Verifier  Test 
Controller  GUI 


r 


Status  of  federates 


Test  Controller 
activities 


Distributed  Simulation  Test  Tools: 
Simulation  Interoperability  Test  Harness 
(SITH) 


•  Supports  development  and  integration  testing  of  HLA  federations 

-  general  purpose  tool  that  allows  federate  emulation  for  runtime 
data  validation,  functional  testing,  and  performance  testing 

•  Built  around  RTI  Verifier  core.  Key  add’l  features: 

-  Ability  to  create  unlimited  stand-in  federates 

-  Object  Script  Creator  (OSC)  for  graphically  creating/modifying 
SDL  test  scripts 

•  The  SITH  uses  a  sophisticated  scripting  capability  to  produce 
complex  data  exchanges,  along  with  a  data  logging  capability,  to 
run  tests  that  lead  to  quick  diagnosis  of  problems 

-  The  SITH  has  been  instrumental  to  the  successful  development 
and  testing  of  several  HLA  federations 


Distributed  Simulation  Test  Tools: 
SITH  (cont’d) 


•  SITH  GUI 
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Distributed  Simulation  Test  Tools: 
SITH  (cont’d) 


•  Object  Script 
Creator  GUI 
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Applicability  of  Simulation  Test  Tools 
to  Net-Centric  Services  Testing 


•  The  following  characteristics  of  the  SITH  and  Verifier  can  be  useful 
applied  to  testing  net-centric  SOAs: 

-  GUI-  this  will  be  necessary  to  control  the  test  environment 
which  the  tool  will  emulate 

-  Test  scripting  capability  (SDL)-  this  will  be  useful  for  setting  up 
and  repeating  parts  of  the  scenario  relating  to  the  network 
response  and  service  behavior 

-  Ability  to  see  the  entity  states  in  the  GUI 

-  Record  entity  state  changes  for  analysis 

-  Run-time  data  validation  capability 

-  Service  or  system  emulation 

-  Network  flooding  capabilities 

•  However,  fundamental  differences  in  underlying  core  services  will 
require  reworking  the  SDL  to  control  net-centric  services 
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SOA  Test:  An  Emerging  Automated  Tool  for 
Testing  SOAs 


•  WSDL  Verification 

-  XML  Validation 

-  Tests  interoperability  against  WS-I  Standards 


•  Unit  Testing 

-  Verifies  web  service  responses  against  valid  and  invalid  data  sets 

•  Data  sets  can  be  composed  of  a  range  of  values  in  legacy  data  stores 
-  E.g.  Microsoft  Excel  or  Database  queries 


Functional  Testing 

-  Scenario  based  testing  using  a  chain  of  services 

-  XML  Databank  used  to  map  the  output  of  a  given  web  service  to 
the  input  of  another 
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SOA  Test:  An  Emerging  Automated  Tool  for 
Testing  SOAs 


•  Scripting 

-  JavaScript,  Jython  (Java-enabled  Python) 


•  Security 

-  Message  layer  security 
•  Username  or  SAML  Tokens 


-  Penetration  testing 


•  SQL  Injections 

-  Passing  SQL  Query  Strings  as  parameters  to  the  Web  Service 

•  Parameter  fuzzing 

-  Unbounded  parameters  leading  to  buffer  overflow  or  explicit  error 
messages 

XML  Encryption  and  Signature 
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SOA  Test:  An  Emerging  Automated  Tool  for 
Testing  SOAs 


•  Regression  testing 

-  Automated  testing  in  continuous  integration  environments 

-  Evaluate  trends  over  time 

•  Load  testing 

-  How  do  multiple  users  affect  timeliness,  content 
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Challenges 


•  Number  and  volatility  of  associated  standards 

-  E.g.  Web  services  with  attachments  must  account  for: 

•  Soap  with  Attachments,  MIME,  DIME,  MTOM  Recommendations 

•  Automated  failover 


•  Federated  registries 

•  Evaluation  of  service  pedigree 
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Findings  and  Recommendations 


*  Significant  overlap  between  HLA  and  GIG  operations 
warrants  closer  look  at  simulation  test  tools  and  approaches 

-  Examined  SITH  due  to  documented  successes  and 
extensible  software  design 


•  Recommend  an  exploratory  prototype  that  reuses  much  of 
SITH  for  net-centric  testing  purposes 

-  Replace  SITH  SDL  with  a  new  scripting  language  that 
leverages  web  services  standards  (WSDL,  BPEL4WS) 


Apply  the  new  SITH-like  application  first  to  interoperability 
testing  of  small  groups  of  services 

-  Then  expand  use  for  performance  and  behavior  testing  of 
larger  groups  of  services  in  more  complex  internet-like 
environments 
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Contact  Us 


Doug  Flournoy 
rflourno@mitre.org 
PSV  (781)271-2774 


Elizabeth  Lee 
elee@mitre.org 

(703)  983-2692 


Rob  Mikula 
rmikula@mitre.org 

(703)  983-7168 
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•  A  current  reality 

•  Program  preparation 

•  Traditional  structured  analysis 

•  UML 

•  The  future 
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How  Should  the  Engineer  Approach 
Unprecedented  Problem  Space? 
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System  Entities 
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A  Single  Model  Will  Not  Work 
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Prepare  the  Enterprise 
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Prepare  the  Enterprise 

•  Select  standards  and  corresponding  templates 

•  Select  preferred  structured  analysis  models 

•  Tailor  templates  for  alignment  with  models  you  choose  to 
use  to  explore  problem  space 

•  Build  data  item  description  (DID)  for  each  specification 
type/model  application 

-  System  specification/TSA 

-  Hardware  performance  specification/TSA 

-  Software  performance  specification/UML 

-  Software  performance  specification/DoDAF 

-  Hardware  ICD/TSA  and  software  IRS/UML  and  DoDAF 

•  Map  organization  and  models  to  template 

•  Apply  SDD  as  a  means  to  capture  models  products 
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3.1 

3.2 

3.2. m 

3.2. m.n 

3.3 

3.3.1 

3.3.1.  m 

3.3.1.  m.n 

3.3.2 

3.3.2. m 

3. 3. 2.  m.n 

3.4 

3.4. m 

3. 4.  m.n 

3.5 

3.6 


Generic  Template 

States  and  modes 

Entity  capabilities 

Entity  capability  m 

Entity  capability  m,  requirements  n 

Interface  requirements 

External  interfaces 

External  interface  m 

External  interface  m,  requirement  n 

Internal  interfaces 

Internal  interface  m 

Internal  interface  m,  requirement  n 

Specialty  engineering  requirements 

Specialty  discipline  m 

Specialty  discipline  m,  requirement  n 

Environmental  requirements 

Precedence  and  criticality  considerations 
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Do  the  Analysis  Work  Using  Preferred 

Models 
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Ultimate  Process  Diagram 
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X:  REFER  TO  PROGRAM  SYSTEM  DEFINITION  DOCUMENT  FOR  EXPANSION 
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The  Process 
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Traditional  Structured  Analysis 


■PROGRESSIVE  FEEDBACK] 


Specification 


NOTE 

Specification  template 
paragraph  numbers  in 
red. 


JOG  SYSTEM  ENGINEERING  10-22-2004 
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Functional  Decomposition 
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TOP 

LEVEL 
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Integrated  RAS  and  N-Square  Diagram 
For  Internal  and  External  Interface 
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Identification  of 
Specialty  Engineering  Constraints 
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ARCHITECTURE-SPECIALTY  ENGINEERING  MATRIX 
(DESIGN  CONSTRAINTS  SCOPING  MATRIX) 


SPECIALTY  ENGINEERING  REQUIREMENTS 
FLOW  INTO  THE  INDICATED  SPECIFICATIONS 
THROUGH  THE  RAS 
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Specialty  Engineering  Allocation 

Specialty  Discipline  Q7  Allocated  to  Architecture  All 


INTERNAL  INTERFACE 
MATRIX  (RED) 


ARCHITECTURE- 
SPECIALTY  ENGINEERING 
MATRIX  (GREEN) 


ARCHITECTURE 

AXIS 

FUNCTION- 
ARCHITECTURE 
MATRIX  (YELLOW) 


FUNCTION  AXIS 


SPECIALTY  ENGINEERING 
DISCIPLINE  AXIS 
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Four  System  Environmental  Classes 
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Three  Environmental  Requirements 

•  System  Layers 

-  Identify  spaces  within  which  the  system  will  have  to  function 

-  Select  standards  covering  those  spaces 

-  For  each  standard,  select  parameters  that  apply 

-  Tailor  the  range  of  selected  parameters 

•  End  item 

-  Build  three  dimensional  model  of  end  items,  physical 
processes,  and  process  environments 

-  Extract  item  environments 

•  Component 

-  Zone  end  item  into  spaces  of  common  environmental 
characteristics 

-  Map  components  to  zones 

-  Components  inherit  zone  environmental  requirements 
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RAS-Complete  In  Table  Form 


MODEL  ENTITY 

REQUIREMENT  ENTITY 

PRODUCT  ENTITY 

DOCUMENT  ENTITY 

MID 
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PID 
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A1 
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H1 1 
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Failure  Rate  <5x10-6 

A12 
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3.1.5 

Reliability 
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A13 
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Mean  Time  to  Repair  <  0.2  Hours 

A1 

Sensor  Subsystem 

3.1.6 

Maintainability 

H12 

Maintainability 

U9R4 

Mean  Time  to  Repair  <  0.4  Hours 
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A1 
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Q 
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Temperature 
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A 
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QX 

Non-Cooperative  Environmental 
Stresses 

A 

Product  System 
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Structured  Analysis  Work  Product 
Capture  and  Configuration  Management 
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Accomplish  the  SW  Analysis 
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Unified  Modeling  Language 
Implementation  Approach 
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Structural  Classifier  Analysis 
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Swim  Lanes  for  Allocation 
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COMPONENT  AX31  COMPONENT  AX32  COMPONENT  AX33 


ACTIVITY  DIAGRAM  FOR  NODE  AX3 
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Progressive  Identification  of  Product 

Entities 
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Dynamic  Diagrams  to  Explore  Product 

Entity  Capabilities 


Sequence  Diagram  State  Diagram  Communication  Diagram 
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The  Entity  Capabilities  Flow  Into  the 
Specification  Through  the  RAS 
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A  Particular  Implementation  Today 
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Publish  the  Results 
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Specification  Review  and  Approval 

Process 
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We  Still  Have  to  Push  These  Together 

Some  More 
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1 .0  Why  Architecture- Based  Systems  Engineering? 
What  does  Architecting  bring  to  the  Systems 
Engineering  Process? 

2.0  DODAF  views  and  Products 

3.0  Four  Aspects  of  Systems  Engineering. 
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Why  Architecture- Based  Systems 

Engineering? 

•  DOD  Systems  Acquisition  paradigm  is  moving  away 
from  acquiring  products  to  acquiring  Capabilities. 

•  Emphasis  on  using  existing  systems  with  new  systems. 

•  Network  Centric  Operations  demands  flexibility  and 
Interoperability  between  System-of-Systems. 

•  Coping  with  Complexity,  uncertainty  and  cost  of 
Engineering  Effective  Systems  and  Enterprises. 

•  Utilizing  Information  Technology  to  improve  Productivity, 
integration,  and  build  unprecedented  Complex  Systems. 
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Architecture-Based  Systems 

Engineering 

•  Responding  to  these  changes  and  challenges:  We  must 
systematically  merge:  Architecting  activities, 

Engineering,  Integration,  and  Evaluation  of  systems. 

•  The  response  is  using  the  Systems  Engineering 
approach  of  iterative  top-down  design  and  bottom-up 
integration  of  software  and  hardware. 

•  DOD  has  directed  that  DODAF  be  used  to  describe 
several  Aspects  of  system  architecture.  DODAF  consists 
of  26  views  products. 

•  DODAF  1 .0  does  not  provide  guidance  for  detailed 
design,  integration,  or  evaluation  of  systems. 

•  DODAF  products  can  be  used  to  create  the  System 
Architecture. 
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Concept  of  Architecture 


•  The  concept  of  architecture  entered  the  domains  of  Software  and 
Systems  Engineering  in  recent  decades 

•  Some  people  argue  that  the  Software  and  System  architecture  are 
similar  to  that  of  civil  architecture 


•  A  building  architecture  is  a  detailed  endeavor  driven  by  the  user  that 
is  meant  to  stay  for  many  years.  The  drawings  are  very  detailed  to 
support  the  builder. 

•  Enterprise  /System  architecture  must  be  more  responsive  to  change 
&  extensible:  flexible,  adaptable,  and  scalable  than  building 
architecture.  It  must  be  interchangeable  /  modular,  &  reusable. 
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Types  of  Architectures 


There  are  two  types  of  architectures: 

(1)  System  Architecture  of  building  real  systems:  Civil, 
Hardware,  and  Embedded  systems  architecture, 

(2)  Domain  Specific  Architectures:  Software,  supply  chain, 
finance,  insurance,  business  processes,  security,  C4ISR, 
avionics,  weapon  system. 

**  Architectures  are  preambles  and  part  of  the  systems 
engineering  process. 
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Architecture  Framework 


•  Framework  is  the  structural  frame  of  reference, 

•  Architecture  Framework  e.g.  DODAF  or 
Zachman  give  the  structure  of  the  architecture. 

•  Framework  does  not  give  detailed  guidance  on 
how  to  design  or  implement  architectures. 

•  Architecture  (Framework  +Design)  =  System 
Architecture 
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DODAF  Linkages  Among  Views 


Operational 

View 


Relates  Systems  and  Characteristics 
to  Operational  Needs 
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View 
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Figure  ES-1.  Linkages  Among  Views 
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System  Architecture  Must  Have: 


•  Every  system/enterprise  must  have  an 
architecture 

•  An  architecture  must  have: 

*  Structure  (Topology)/Organization, 

*  Functionalities:  Function  Blocks,  Modules 

*  Connectivity:  Linkages  and  interfaces- 
Function  Blocks  Diagram  for  a  given  set 
of  events 
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Architecture-Based  System  Design 

Process 

•  Two  approaches: 

(1)  Structured  Analysis  (SA): 

using  IDEFO,  IDEFIx,  DFD,  FFBD, 
EFFBD 

(2)  Object  Oriented  (00): 
using  UML  or  equivalent 
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Systems  Engineering 

Systems  engineering  is  a  multidisciplinary  subject 
dealing  with  the  integration  of  all  parts  of  a  system 
(hardware,  software,  and  operator)  into  the  real  world 
environment.  It  provides  rationalization  for  tradeoffs  in 

meeting  the  requirements  and  building  the  system. 
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Systems  Engineering  Process 


Systems  Engineering  Process  deals  with  all  aspects  of  system  design, 
development,  integration,  evaluation,  disposal,  and  management. 

(1)  Determination  of  requirements, 

(2)  Definition  of  system  architectures, 

(3)  Establishing  performance,  reliability,  and  availability  goals 

(4)  Tradeoff  analysis:  performance  versus  cost,  reliability,  support, 

(5)  Decomposition  and  partitioning  of  system,  and  system  design, 

(6)  Integration  of  building  blocks,  interfaces,  testing,  and  safety, 

(7)  Metrics  to  evaluate  all  these  activities,  evaluate  the  effectiveness 
of  alternatives  and  options, 

(8)  System  engineering  management:  system  acquisition,  project 
management,  and  risk  analysis. 
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Partitioning  Levels  of  System  Architecture 


There  are  five  levels  of  architectures: 

(1)  Svstem/Enterprise  level - ►  Conceptual  Architecture 

Expressing  stakeholders  needs,  concerns,  capability  requirements,  and 
strategy 

(2)  Subsystem  level  - >  Functional/Logical  Architecture 

Performance,  functional  connectivity,  and  interfaces 

(3)  Module  level  - ►  Physical  Architecture 

Physical  connectivity  and  interfaces 

(4)  Function  Block  level _ >  Operational/Implementation  Architecture 

(5)  Subfunction  level - *  Bottom-up  Integration  and  Interfaces 

Architecture 

*  Physical  and  functional  connectivity  are  interfaces  driven 
**  Functional  and  Physical  architectures  are  performed  concurrently 
**  The  final  architecture  is  achieved  by  iteration  between  these  levels 

**  These  architectures  evolved  into  architecture  Views:  System,  Operational, 
Technical  and  All 
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System  Architectures  Flow  and 


Relationship 


ARH 
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Conceptual  Architecture  Development 

Process 


Design 

Changes 


a 


Stakeholders: 

*Requirements 

*Needs 


Develop 

Operational 

Concept 

‘Concerns 

‘Scenarios 

w 

Operational  Concept 

DODAF:AV-1 , 
OV1 ,  OV-2, 
OV3,SV-1 


>• 


Develop 

Feasible 

Requirement 


System 

Requirements 


Requirements 
Changes,  Proof-of- 
Concept  experiments 


Operational 

Architecture 

Iteration  between  the  development  levels 


System  Analysis 
AOA 
COEA 

Cost-Effectiveness 

Proof-of-Concept 


Conceptual 

Architecture, 

Scenario 


Functional  Architecture  Development  Process 

Conceptual  Architecture  And  System  Requirements 


'\r 


Operational 

Concept 


Generate  Functionalities 
From  Operational  Concept 


Functionalities! 

List 


Functional 
Requirement 
Inputs  and  Output 


DODAF:  AV-1, 

OV-1 ,2,3, 

SV-1 ,2, 4, TV-1 

Preliminary  _ 

Physical  Architecture 


Functional 

Architecture 

Changes 


Functional  and 
Data  Models 

Functional  Decomposition 
Using  IDEFO  (/ntegrated  Definitj^ 
for  Function  Modeling)0 


Functional 
Model 


Boundary 
Inputs,  Control 
And  Objectives 

Input/Output 

Requirement 


Architecture 

Issues 


Determine  the 
Relationship  among 
Inputs/Outputs 
Establish  Data  Model 


Map  Input/Output 
To  Requirements 
Integrate 
Functional  &  Data  Modelk 


Functional  Architecture' 


Physical/Structural  Architecture 


•  The  physical  architecture  gives  the  physical  resources  to  perform 
the  system  functions.  It  is  developed  concurrently  with  the 
Functional  Architecture. 

•  It  identifies  resources  to  form  the  structural  architecture.  Physics- 
Based  simulations  provide  insights  into  the  physical  architecture 

•  Physical  architecture  is  a  description  of  the  partitioned  elements  of 
the  system  without  their  performance  specifications. 

•  It  accounts  for  all  the  nonfunctional  attributes:  Reliability,  availability, 
security,  scalability,  reusability. 
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Physical  Architecture  Development  Process 
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Physical  Architecture  of  a  Weapon  System 
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Reliability/Availability  Block 

Diagram 

Example  of  a  Physical/Structural  architecture 


Series  parallel  Architecture 
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Development  Process  of  Integration  and 

Interfaces  Architecture 
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Development  of  Operational  Architecture 
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Systems  Engineering  Process 
Seven  Phases  for  Acquisition/Production 
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Systems  Engineering  Interactive  Model 
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Software  Architecture 


•  Software  architecture  deals  with  the  design  and 
implementation  of  the  high-level  structure  of  software. 

•  A  model  of  five  views  called  (4+1 )  is  used  to  describe  the 
software  architecture: 

(1)  Logical  View, 

(2)  Process  View, 

(3)  Development  View, 

(4)  Physical  View 

(5)  Use  Case  view 
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Software  Architecture  Model 

Four  +  One  View  Model 


End-user 

Functionality 


Evaluation  Programmers 

Verification  Software  Management 

Testing 


Logical  View 


Use  Cases 
Scenarios 


Integrators 
Performance 
Scalability 


ir  \ 

Process  View 

Physical  View 

- ► 

Systems 

Engineers 

Communications 

Topology 


ARH 


31 


Merging  Architecting  Into  the 
Systems  Engineering  Process 
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Architect  and  Systems  Engineer 

Spaces 
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Level  of  Architecting  Involvement  in 
the  Systems  Engineering  Process 
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System  Integration  and  Interfaces: 

Why  is  it  important? 

•  System  Integration  is  a  critical  part  of  the  Systems 
Engineering  Process,  it  brings  the  decompositions 
together. 

•  Interfaces  are  the  keys  to  a  successful  Integration. 

•  The  recognition  and  identification  of  Interfaces  are  very 
important  in  building  responsive,  robust,  and  Effective 
systems. 

•  Interfaces  occur  at  the  boundaries  of  the  building  Blocks 
of  the  system:  functionally,  physically,  and  operationally. 

•  The  capability  of  recognizing  and  counting  the  interfaces 
will  ensure  the  success  of  the  architecting  and 
engineering  process. 
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The  Binary  Interfaces  Model 

•  We  build  systems  to  give  outcomes.  System  behavior  is  described 
by  its  outcomes. 

•  We  will  use  a  binary  building  blocks  model  to  determine  and  count 
interfaces. 

•  Given  a  system  whose  functionality  is  achieved  by  integrating 
(merging)  n  =  three  building  blocks:  A,  B,  C. 

•  The  number  of  possible  outcomes  is  “2n”=23=  8. 

•  The  outcomes  are  represented  by: 

(1 )  Events  (Euler,  or  logical)  Diagrams, 

(2)  Network  (  Decision  tree), 

(3)  Truth  table, 

(4)  Combinatorial  mathematics. 
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Binary  Model  of  Interfaces 

•  When  the  functionalities  of  A,B,  and  C  merge  new  functions  are 
created  and  are  represented  by  the  events  (logic)  diagram 


Interfaces  occur  when  two  or  more  functionalities  are  merged,  we  call  them 
mathematical  intersections  of  order  two  or  more.  Fo£this^xample: 

There  are  three  second  order  interfaces:  ABC,  ABC,  ABC 

and  a  third  order  interface:  ABC 
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Subsumed  Interfaces 


•  As  we  integrate  systems  not  all  interfaces  are 
recognized  and  accounted  for. 

•  The  question  is  what  happens  when  interfaces 
are  subsumed? 

•  Subsumed  interfaces  are  either  absorbed  by 
other  interfaces,  or  they  are  neglected.  This  can 
happen  when  functionalities  are  not  merged 
properly,  poor  coupling,  low  amplification,  or  by 

organizational  design. 

•  The  impact  of  subsumed  interfaces  is  serious 
degradation  in  system  effectiveness. 
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Logically  Integrated  System  versus 
System  with  Subsumed  Interfaces 


The  area  of  the  function  is  proportional  to  its:  Design  Limitations -Design  Adequacy 
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Comparing  Two  Systems 

System  S-, with  subsumed  Interfaces  System  S2  Logically  Integrated 
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Conclusion 


•  Architecting  is  a  major  contributor  to  the  engineering  of  responsive 
systems.  It  strengthens  the  systems  engineering  process  and 
provides  critical  insights  into  the  stakeholder  and  user  requirements. 

•  System  architecture  and  software  architecture  should  be  congruent. 

*  Interfaces  are  the  keystones  of  system  integration. 

*  Merging  the  architecting,  engineering,  integration,  and  evaluation 
into  the  Systems  Engineering  process  will  provide  a  comprehensive 
and  a  balanced  approach  to  deal  with  the  complexity  of  building 
effective  systems. 


Architecture- Based  Systems 
Engineering  and  Integration 


Backup 
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Next  Generation  Systems 


•  Traditional  Systems  Engineering  and  Systems  Acquisition  are 
focused  on  Vertical  (Stovepipe)  System  design. 

•  System  acquisition  have  moved  away  from  the  traditional  model  to 
acquiring  Capabilities. 

•  Next  generation  systems  must  be  flexible,  scalable,  interoperable, 
and  adaptable/agile.  Therefore,  the  design  of  Future  Systems 
should  be  based  on  “Capability  Building  Blocks” 

•  Current  and  Future  Network  Centric  Operations  requirements 
mandate  Interoperability. 

•  Current  and  future  systems  are  software  intensive.  Software  is 
embedded  into  systems  and  intermingled  within  System-of-Systems. 

**  We  need  a  comprehensive  approach  to  build  systems,  whose 
architectures  are  compatible  with  the  systems  engineering  process. 
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System  Architecture  Model 


The  system  architecture  model  is  the  model  of  function 
and  form  that  defines  the  physical  structure,  functional 
flow,  interfaces,  and  the  connectivity  of  the  system 
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Architecture  Implementation 

•  Architecture  Implementation  Principles 

•  Architecture  framework 

-  DoDAF 

-  Zachman 

•  Implementation  methodology  of  software 

-  SOA 

-  Data  driven 

-  Client/Server 

-  Model  driven  architecture  (MDA)  using  UML 

-  Event  driven  architecture  (EDA) 

-  Component-based  architecture 

-  Aspect  Driven  Architecture 

-  Domain  Driven  Architecture 
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Input/Output  of  System  Design  Activities 
During  Architecting  and  Engineering 


Design  Activity 

Input 

Output 

(1)  Conceptual  System  Design 

Stakeholders 

Needs  &  Concerns 
Operational  Concept 

Requirements 

Conceptual  Architecture 

Concept  of  Operation 

(2)  Functional  System  Design 

Requirements 

Conceptual  Architecture 

Functional  Architecture 

System  Level  Specification 

(3)  Physical  System  Design 

Requirements 

Conceptual/Functional 

Architecture 

Physical  Architecture 

System  Level  Specification 
Performance  Specification 

(4)  Integration  &  Interfaces 

Subsystem,  Module, 

Function  Block 

Functional  &  Physical 
Architecture 

Performance  Specification 

Integration  &  Interfaces 
Architecture 

Design  Specification 

KPPs 

(5)  Operational  System  Design 

And  Risk  analysis 

Requirements 

Functional  Architecture 
Physical  Architecture 
Interfaces 

Operational  Architecture 
Performance  Specification 

Risk  management 

(6)  Evaluation  and  Testing 
of  system  design 

Integration  &lnTerfaces  & 
Operational  Architecture; 

Design  Parameters  46 

Tradeoff  &Feedback 

Conceptual  Architecture 


•  The  conceptual  architecture  is  a  high  level  abstraction  of  the 
requirements. 

Virtual  Simulation  provides  insights  into  the  conceptual  architecture. 

•  It  is  obtained  by  partitioning  the  requirements  based  on  the 
stakeholders  needs,  concerns,  and  requirements. 

•  The  conceptual  architecture  establishes  the  system/enterprise 
design  requirements  based  on:  Input/Output,  Performance  and  cost 
tradeoffs,  and  mission  analysis. 

•  Proof-of-concept  experiments  are  used  to  support  the  Conceptual 
Architecture. 

•  In  software  engineering  conceptual  architecture  can  be  derived  from 
“use  cases”,  scenarios,  and  activity  diagrams. 
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Functional/Logical  Architecture 


•  The  functional  architecture  contains  the  system  functions,  configuration 
units,  building  blocks  and  components.  It  is  represented  by  a  directed 

networl . 


•  It  is  a  logical  representation  of  what  the  system  must  do  ?,  it  provides  a 
decomposition  of  the  system  objective.  Physics-based  simulations  give 
insights  into  the  functional  architecture 

•  A  logical  model  of  transforming  inputs  into  outputs  using  control  information- 
flow  throughout  the  functional  decomposition.  It  defines  the  unctions  and 
the  data  flows. 


•  It  is  derived  using  an  IDEFO  model,  FFBD,  DFD,  or  “00”  UML  methodology. 

**  Functions  express  activities,  actions 

***  Form  follows  function,  Physical  follows  functional 

***  Fit  follows  form  which  follows  function:  "unction,  Form,  Fit 
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Functional  Decomposition 

•  Several  system  features  are  used  to  partition  a  system  into  building 
blocks 

(1)  At  the  top  level  systems  operate  in  modes.  Modes  lead  to  partition 
the  system  into  subsystems. 

(2)  Each  subsystem  is  partitioned  into  modules  based  on  grouping  of 
functions. 

(3)  Modules  that  have  multiple  inputs  and  outputs  are  partitioned  by 
tracing  the  input  to  the  output  to  establish  the  function. 

(4)  Partition  each  function  block  into  subfunctions:  Circuits/Applications. 

(5)  The  lowest  level  of  the  functional  architecture  identifies  devices  and 
activities  such  as,  transmit  output,  receive  input,  store  output,  format 
input,  amplify  input. 
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Relationship  Among  DODAF  Views 


The  operational  view  provides  the  information  exchanges, 
interoperability  levels,  and  performance  parameters  required  to 
support  a  mission  or  task 

The  systems  view  defines  system  attributes,  and  provides  the 
basis  for  comparing  system  performance  against  operational 
requirements. 

The  technical  view  defines  the  standards  to  the  implementation 
criteria  for  fielding  an  interoperable  system. 

The  three  views  and  their  interrelationships  are  designed  for 
deriving  measures  such  as  interoperability,  performance  and 
measuring  the  impact  of  these  metrics  on  operational  mission 
and  task  effectiveness 

Many  data  elements  of  the  products  are  used  in  more  than  one 
product  and  there  are  several  mapping  relationships  between 
products. 
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Integration  and  Interfaces  Architecture 


•  Integration  is  a  bottom-up  process  of  assembling  the  system  from  its 
components.  Engineering-Based  simulations  and  mockups  provide  insights 
into  the  integration  and  interfaces  architecture. 

•  Integration  is  a  hierarchical  process.  At  the  top  level,  it  is  the  merging  of 
subsystems  functionally  and  physically.  At  the  subsystem  level  it  is  the 
merging  of  modules  functionally  and  structurally. 

•  Integration  and  Interfaces  Architecture  merges  the  functional,  physical,  and 
conceptual  architectures. 

•  Interfaces  are  critical  parts  of  the  systems. 

•  Interoperability  is  based  on  system-to  system  interfaces;  it  is  a  major  driver 
in  architecting  SoS&  FoS. 
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Operational  Architecture 


•  The  operational  architecture  brings  together  through  scenarios  the 
requirements  decomposition  (conceptual  architecture)  with  the  functional, 
physical,  and  integration  ana  interfaces  architectures. 

•  Scenarios  are  used  to  develop  the  operational  architecture.  It  identifies  the 
system’s  internal  and  external  interfaces,  connectivity,  and  nodes. 

•  It  provides  suf  icient  details  to  evaluate  system  performance,  performance 
tradeoff  analysis,  and  risk  analysis. 

•  Operational  architecture  describes  the  mission,  tasks,  operational 
elements,  and  information  flows  required  to  accomplish  or  support  system 
functions 


•  It  is  concerned  with  interoperability  and  human  interfaces.  It  defines  and 
analyzes  functional  activation  and  control  structure.  Engineering-Based 
Simulation  provides  insights  into  the  operational  architecture. 
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Integration  and  Interfaces  Architecture 

Continue 


•  Interfaces  are  connections  for  tying  the  system  parts  to  each  other,  and 

creating  new  functions  and  nodes. 

•  There  are  internal  interfaces  of  merging  one  system  part  to  another. 
External  interface  is  the  connection  of  the  system  to  another  system- 
interoperability. 

•  Integration  and  interfaces  are  united  activities  in  building  systems. 

•  Interfaces  are  the  basic  building  blocks  of  the  system  integration  process. 

•  There  are  several  types  of  interfaces:  Electrical,  logical,  physical, 
environmental,  communications,  thermal,  social  (human  interfaces). 

Interfaces  are  the  mathematical  intersections  of  the  interacting  functions 
that  create  new  functions,  or  activities. 
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Products  of  Architecture  Views 


•  Each  view  is  described  by  a  set  of  products,  e.g., 
graphics,  diagrams,  tables,  scripts. 

•  Each  product  contains  “data”  that  describes  some  aspect 

of  the  architecture,  e.g.,  functionality,  structure, 
connectivity. 

Issues: 

(1)  Are  the  products  consistent? 

(2)  What  are  the  relationships  among  the  products? 

(3)  Does  the  group  of  products  contain  all  the  information 
that  define  an  architecture? 
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Products  of  Architecting 

•  Architecting  is  a  key  upfront  activity  and  is  strongly  tied  to  the  customer  and  user 
(stakeholder). 

•  Architecting  is  a  major  part  of  the  definition  and  design  phase  of  the  Systems 
Engineering  Process 

•  Architecting  Products  To  Systems  Engineering 

(1) Provide  visualization  of  needed  capability  based  on  requirements 

(2) System  Conceptualization  and  Innovation 

(3)  Interfacing  with  stakeholders 

(4)  Provide  Architecture  Framework:  DODAF 

(5) Generate  and  balance  operational  views  and  systems  views 

(6)  Identify  Standards 

(7) Provide  capabilities  road-map 

(8) System  partitioning:  functional,  Physical,  Logical,  and  Operational 

(9) Define  System  boundaries 

(10) ldentify  tradeoffs  space:  Cost,  Capability,  Support,  Reliability,  availability,  and  Risk 
(1 1  identification  of  interfaces  (external  and  internal) 

(12)Modeling/operation  analysis 
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Products  of  Systems  Engineering 

System  Definition,  Design,  Development/Production,  Deployment,  and 
Disposal. 

(1) Visualization  of  what  a  system  should  look  like:  Engineering  the  System 

(2) Tradeoff  studies  -  AOA  -  CE,  CB,  Risk 

(3) Requirements  decomposition/Functional  Decomposition,  traceability 

(4) System  and  design  specification,  and  Configuration  Management 

(5) Detailed  system  analysis  and  identification  of  capability  drivers 

(6) Detailed  system  partitioning 

(7) Detailed  System  integration  and  interfaces;  codesign  issues 

(8) System  Acquisition 

(9)  Resource  allocation  (scope) 

(10)  Integrated  Logistics  Support  (ILS) 

(11) Select  metrics:  MOEs,  MOPs 

(12) Data  exchange  and  collaboration 

(13) Test  and  Evaluation  Plan 

(14)  Management  of  the  Systems  Engineering  Process 
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Systems  Engineering  Attributes 


•  Systems  Engineering  is  the  integrating  mechanism  across  the 
technical  efforts  related  to  the  design,  development,  acquisition, 
integration,  manufacturing,  verification,  deployment,  operations, 
support,  disposal,  and  user  training  of  systems  and  their  life  cycle 
processes. 

•  Systems  Engineering  develops  technical  information  to  support  the 
program  management  decision-making  process  in  meeting:  Cost, 
Schedule,  Performance  and  Risk. 
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ALL  Views 
Two  Views 


Framework 

View 

Product  Name 

General  Description 

AV-1 

Overview  and 
Summary 
Information 

Purpose,  scope,  stakeholders, 
user,  environment  description, 
analytical  assessment 

AV-2 

Integrated 

Dictionary 

Data  repository  with 
definitions  of  terms  used 
in  all  products 
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Operational  View 
Nine  Views 


Framework 

Product 

Product  Name 

General  Description 

OV-1 

Graphic  high  Level 
Operational  Concept 

High-level  graphical  &  textual  description  of  operational  concept 

OV-2 

Operational  node 
Connectivity  Description 

Operational  nodes,  operational  activities  at  each  node, 
connectivity  and  information  exchange  need-lines  between  nodes 

OV-3 

Operational  Information 
Exchange  Matrix 

Information  exchanged  between  nodes,  and  the  relevant 
attributes  of  that  exchange 

OV-4 

Organizational 
Relationship  Chart 

Operational  role,  or  other  relationships  among 
oraanizations 

OV-5 

Operational  Activity 
Model 

Capabilities,  operational  activities,  relationships 
among  activities,  inputs,  and  outputs.  May  show  cost. 

OV-6a 

Operational  Rules 
Model 

Describes  sequence  and  timing  of  operational  activities  - 

identifies  business  rules  that  constrain  operation 

OV-6b 

Operational  State 
Transition  Description 

Describes  sequence  and  timing  of  operational  activities 
identifies  business  process  response  to  events 

OV-6c 

Operational  Event 
Trace  Description 

Describes  sequence  and  timing  of  operational  activities 

traces  actions  in  a  scenario  or  sequence  of  events 

OV-7 

Logical  Data  Model 

Data  requirements  documentation  and  structural  business 
process  rules  of  the  Operational  View 

Systems  View 
Seven  Views 


Framework 

Product 

Product  Name 

General  Description 

SV-1 

Systems  Interface 

Description 

Identification  of  systems  nodes,  systems,  items,  and 
their  interconnections  within  and  between  nodes 

SV-2 

Description  of  Systems 
Communications 

Systems  nodes,  systems,  and  system  items  and 
their  related  communications  lay-downs 

SV-3 

Systems-Systems 

Matrix 

Relationships  among  systems,  system-type  interfaces 

Planned  vs.  existing  interfaces 

SV-4 

Systems  Functionality 

Description 

Systems  Functions  and  data  flow  among  them 

SV-5 

Operational  Activity  to 
Systems  Function 
Traceability  Matrix 

Mapping  systems  to  capabilities,  functions  and 
Operational  activities 

SV-6 

Systems  Data 

Exchange  Matrix 

Systems  data  elements  exchanged  between  systems 
and  the  attributes  of  that  exchange 

SV-7 

Systems  Performance 
Parameters  Matrix 

Performance  characteristics  of  elements  for  a 
given  timeframes) 
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Systems  View 
Six  Views 


Framework 

Product 

View  Name 

General  Description 

SV-8 

Systems  Evolution 
Description 

Planned  migration  of  systems  to  more  efficient  suite,  or 
Evolving  a  current  system  to  future  implementation 

SV-9 

Systems  Technology 
Forecast 

Time  table  of  emerging  software  &  hardware  products 
Technologies  and  that  will  impact  future  architecture 

SV-10a 

Systems  Rules 

Model 

Describes  systems  functionality  constraints  due 
to  some  aspect  of  systems  design  or  implementation 

SV-10b 

Systems  State 
Transition  Description 

Describes  systems  functionality:  identifies  responses  of 
a  system  to  events 

SV-10c 

Systems  Events 

Trace  Description 

Describes  systems  functionality:  identifies  system  changes 
due  to  critical  sequences  of  events  of  the  Operational  View 

SV-11 

Physical  Schema 

Physical  implementation  of  the  Logical  Data  Model 
entities:  message  formats,  file  structure,  physical  schema 
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Technical  Standards  View  Products 

Two  Products 


Framework 

Product 

Product  Name 

General  Description 

TV-1 

Technical  Standards  Profile 

Listing  of  standards  that  apply  to 

System  View  elements  in  a  given 
architecture 

TV-2 

Technical  Standards 
Forecast 

Description  of  emerging  standards  and 
potential  impact  on  current  System  View 
Elements  within  a  set  of  timeframes 
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Tailoring  the  Views 


•  Not  all  software  architectures  need  all  five  views.  Not 
relevant  views  can  be  omitted  from  the  architecture 
description.  For  example,  the  physical  view  can  be 
omitted  if  there  is  only  one  processor,  also,  the  process 
view  can  be  omitted  if  there  is  only  one  program.  For 
very  small  system  it  is  possible  to  merge  the  logical  with 
the  development  view. 

•  Similarly,  in  the  DODAF  not  all  views  are  essential.  In 
general,  AV-1  ,AV-2,  Ov-1  ,OV-2,OV-3,  SV-1 ,  and  SV-4 

are  essential.  The  remaining  products  are  supporting 
views. 
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Logical  Software  Architecture  & 
Process  Software  Architecture 


•  The  Logical  Architecture  supports  the  functional 
requirements;  it  describes  the  “Object  Oriented”  system 
decomposition  using:  Class,  Collaboration  diagrams  and 
the  sequence  diagram. 

•  The  Process  Architecture  gives  Process  partitioning  into 
Tasks;  it  describes  how  processes  communicate  and 
interact.  It  accounts  for  the  nonfunctional  attributes: 

performance,  availability,  reliability,  security,  and 
scalability.  Activity  diagrams  are  used  to  describe  this 
view 
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Development  Software  Architecture 


•  The  Development  Architecture  describes  the  actual 
software  modules  of  the  system  and  its  interaction  with 
the  environment.  Packages,  subsystems,  and  class 
libraries  are  considered  modules. 

•  The  Development  view  gives  the  layers  of  the  system: 

(1)  User  Interface  layer, 

(2)  Presentation  layer, 

(3)  Application  Logic, 

(4)  Business  Logic. 

•  The  Development  Architecture  is  given  by  module  and 

subsystem  diagrams. 
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Physical  Software  Architecture 


•  The  }hysical  Architecture  is  a  mapping  of  Software  onto 
the  Hardware.  It  describes  how  the  applications  are 
installed  into  the  hardware,  and  how  it  executes  in  a 
network  environment.  It  takes  into  account  the  non¬ 
functional  requirements  of  the  system:  reliability, 
performance,  availability,  reuse,  scalability.  Deployment 
diagrams  are  used  to  represent  the  Physical 
Architecture. 
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The  “Plus  One”  View 
Use  Case  Architecture 

•  The  Use  Case  View  is  putting  it  all  together.  The 
elements  in  the  four  views  are  shown  to  work  together  by 
using  a  set  of  scenarios.  The  scenarios  are  abstractions 
of  the  most  important  functionalities  (requirements). 
Scenarios  are  expressed  by  using  Objects  and 
Interaction  diagrams 
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General  formula  of  Interfaces 


The  total  number  of  interfaces:  (  2n  -  n  -  1 ) 


n 


The  number  of  second  order  interfaces:  02  (n-2)i  2 


n! 


The  number  of  third  order  interfaces  : 


n 

c 


n! 


n(-3) !  3! 


The  total  number  of  possible  outcomes: 


n 


** 


2"=ia 

m=0 

A  System  that  contains  all  intersections  is  called  Logically  Integrated 
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’  Digital  DatelVtenagemertt-  ’ — 

An  -Update - 

NEftA  Systems  jgngineeijing  Conference 
San  Diego,  California  -f  October  2005 

Cynthia  C.  Hauer 

Millennium  Data  Management,  Incorporated 

Huntsville,  Alabama 


■  Looking  back 

■  Framing  the  issues 

■  Elements  of  the 
solution 

S  Cause  a  rid- effect 

■  Pafiaeea-versus- 
process 

■\The  way  forward 


Data' management  is  trie  structured  processes  arid 

systerfislmalplan  TorJ  acquire  ana  maintain  data] 
consistent  witfrreqmrements,  tnrougnout  tne 
patalite  cycled 


Identification/Definition 

•  Data  Requirements,  Life  cycle  Needs 

Preparation 

•  Internal  and  External  Data 

Control 

•  Document  control  processes 

•  Import/Export  of  data 

•  IP,  proprietary,  limited  access 

•  llJser  aSthorizatlons,  use  requests 

— •  Master  Lists  - 

•  Data  MarlUng 

Dispositioning 

•  DMP 

•  Delivery  (digital  or  physical) 

Archival 

•  Project  files,  decision  data,  data  retention 


Changes  were 


in  me  approacn 
range,  and 
the  methods  tor  the 
DlVJ  functions 


Looking  back 


DM  was  mandated  in  the  1960's  by 
Congress  and  the  DoD 

Original  intent 

*  Eliminate  redundancy  of  dala  production 


and  storage 

“s“Ce ptfal ize  'systems  use 

•  Acquire  onlly  spec  fic  data  for  deliberate, 
planned,  and  specified  use  /  /  /  / 


trategic  Planning,  Continuing  Coals,  Increased  Urgency 


■  DM  evolved  out  of  DoD  and  Congressional  concerns  (circa  1960) 

•  $$$  being  spent  on  technical  data  and  no  one  knew  how  much 

•  Available  $  was  not  being  spent  effectively 

•  Much  of  what  was  being  acquired  was  unnecessary,  redundant,  and 
obsolete 

•  DM  was  an  expensive  resource  that  was  not  being  managed  wisely 

•  Finally  -  no  one  knew  what  was  being  purchased,  ordered,  or  used, 
anyway 

■  Industry  complaints  increased  re:  the  number  and  variety  of 
management  systems  bdlnglmposed  on  them  fh  development  ana 
production  contracts! 

•  Data  was  pushing  up  the  cost  of  contracts 

d  Perception  was  that  DoD  was  attempting  to  increase  and  maintain 
HBent^nsf-^nt^ets^y-reg-u-i-H-Rg-even-mo-re-i-nf-e^matHon 

•  And  no  one  knew  whatlhe  originally  acquired  data  really  was,  what  m 
supported,  and  what  anyone  was  doing  with  it 

■  Perception!  the  necessity  for  some  ofj  the  management  systems 
and  the  s|pporting*data  was  questionable  and  Unclear 

•  Some  elements  required  and  acquiredlthe  same  things  in  different 
ways 

•  Some  DoD  systems  required  generation  of  data  that  was  already 
available  in  the  contractor's  own  internal  data  systems 


So  What  Has  Changed? 


More  data  than  ever 
Less  money  than  before 
Business-Mode^-  revomtio' 


-Jransaction^and  decision  timeJs 
compressed 

Network' centric  strategies 


And  "technology"  is  not  the  solution 


Vertical  Integration 


Business  Relationships 

. 


DoD  Dfesign  Bureaus 


(7. 


Build  a  system 
that  can  meet 
these  KPPs. 


Weapon  System 
Acquisition  I  rocess 


Inc 


ABC 


r: 


V) 


Virtual  Co..  Inc. 


dUdUl 


Digital  Data  Environment 


ree  Primary  Areas  -  Principles,  Practices,  and  Training 


•  Basic  tenets 
_an_d  values 

•  General 


•  Implementation 
specifics 

•  Organization-specific 


rrairnnc 


Basic  and  Advanced 
(PM  and  Practitioner) 


r /rii  jtjj  jtj 


Data  Management  “Solution:”  contlued 

Three  Primary  AreasH  Principles,  Practices,  and  Training 


Pm\v\p\3 


J  .  ,  J 


*  Implementation 
specifics 

•  Some  are 
organizational- 


ferripJate 


\NS1 


Bas 

(Pl\, 


Data  Management  “Solution,”  conHed 

Three  Primary  Areas  -  Principles,  Practices,  and  Training 


DoD  5010. 12-M  > 

DoD  4140.1  -R 

Industry  implementation  guides 

FAR 

DFARS 

Service-specific  instructions 
LJDEF  and  EIA927,  EIA836 


^Implementation 
^specifics 
•  Some  are 
)/ganizational- 
specific 


Partially  obsolete 
Partially  in  conflict 
Missing  elements 
Potential  solutions 


TrsijJjjjLk 


Basic  and  Advanced 
(PM  and  Practitioner) 


h  HI 


I  ■ 


JJ 

1  3 


continued 


Three  Primary  Areas  -  Principles,  Practices,  an( 


Pil.fJDJpJB: 


•  Basic  tenets 
_and  values 

•  General 


ractices 


ANSI  8: 
HB  859 


859 


fsrnplatss^ 


DoD  5010. 12-M  > 

DoD  4140. 1-R 

Industry  implementation  guides 

FAR 

DFARS 

Service-specific  instructions 
JJDEF  and  EIA927,  EIA836 


Implementation 
specifics 
•  Some  are 
uganization-  specific 


Best  Pr&ctic 


Career  Field  (DAU) 
Colleges  and  Universities 


mpnNPi 


Basic  and  Advanced 

(PM  and  Practitioner) 


Data  Management  “Sojution,”  continued 

Three  Primary  Areas  -  Principles,  Practices,  and  Training 


DoD  5010. 12-M 
DoD  4140.1  -R 

Industry  implementation  guides 

FAR 

DFARS 

Service-specific  instructions 
UDEF  and  EIA  927,  EIA  836 


•  Implementation 
specifics 
■*-  Some-are- 
organization- 


Basic  tenets 


and.  values 


General 


specific 


DAU 

Private  Academia 


Basic  and  Advance* 

(PM  and  Practitioner) 


Data  Management  “Solution,”  co 

Three  Primary  Areas  -  Principles,  Practices,  and  1 


nued 
■  ■ 

lining 


Prinv]p\3s 


•  Basic  tenets 
.and  values 

•  General 


•  Wasting  more  money,  on  more  data  that  is  not  required, 
and  driving  upl  costs,  as  welHWMWBMWBMM 

Allowed  DM  to  make  a  contribution,  not 

_»_Contifl4Bng-to*g-nore-t-h€-obv-ious-problams-f-04l-alJ-parties- 

Enabled fiaradiqmalflfc  changes  in  "acqlisBon"^^ 
afld-extended-tnem-  to  "logist-ios^-not - 

•  Ignoring  me  obvious  cause  and  effect  between  data  and 
end-ittem  costs  in  contracting 

•  Discouraging  commercial  suppliers  from  contributing 
high-end,  niche  end| items 


•  Allowing  contractors  to  do  business  as  usual 


i_Crafted_an^accepted  a  nd_va  I  ue -added 
role 

i  Built  credibility  and  acceptance 


Process  first!  then  automation 

SeqBenomg  "ule—  pBsh/pPlP^^rrectiy 

Creating  intentional,  understood 
ojltcomfes 

Strateg  ijc^  th  i  n  king 

•  Enterprise  focus 


■  Consistency 

•  Expectations  and  practice 

■  Communication 

4  Internal  anil  external 

■  Training 

•  Practitioners  and  other  disciplines 

«_Syn4h4^flj^i-ng-U-SG-aflcl-iflcl4JSt-ry — 

\  effectively 

m'  Finding  our  way  towafds  rejinteg fating 
with  associated  domains 


■  DM  goals,  need,  and  requirements  are  the 
same  cis  they  weife ’irp  1960  1 

■  Urgency  to  transition  DM  to  the  digital 

/  envfrorfffTent  increased  I  |  1  ' 

•  And  we  met  the  challenge 

S-  Recognized  pat-bway-|aoo-i  mporaafleje-f-oc — 
*pFefeo(i|tiori|  identified  andlfouifid 

m. AN  SI  - 8 59 Is  tne  begi|irling 

■  Handbook  859  is  complete 

■  DAU  courseware  development  is  underway 


Implementing  Systems  Engineering  Processes  to 
Balance  Cost  and  Technical  Performance 


Dr.  Mary  Anne  Herndon 
Transdyne  Corporation 


Sandra  Salars 
MEI  Technologies 


October  26,  2005 


Dr.  Mary  Anne  Herndon 
858-271-1615 


Sandra  Salars 
281-283-6182 
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Continuous  and  Effective  Program  Performance  Sustainment 
in  Multi-Year  System  Engineering  Programs 


6  Critical  Program  Performance  Challenges.... 


Large  scope  technical  services  and  products  programs  are  intended  to  provide  wide 
range  of  service  levels  and  product  maintenance  support  over  5-10  years. 

Sustaining  multi-year  technical  service  and  product  support  levels  is  impacted  by 
increases  in  costs,  staff  transitions  and  changing  customer  requirements. 

a-  Candidate  Engineering  Solution . 

Sustaining  service  levels  and  technical  performance  at  planned  costs  is  facilitated  by 
applying  SE  Vee  life  cycle  model  to  engineer  a  support  framework  across  the  program. 

This  support  framework  is  used  to  balance  the  costs  of  maintaining  program  infrastructure 
functions  and  technical  resources  with  the  costs  of  achieving  program  performance  goals. 

Application  of  the  practices  in  the  CMMI®  Process  Areas  (PAs)  are  used  across  the 
program  and  projects  to  implement  the  relevant  phases  in  the  SE  Vee  model. 
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Challenges  in  Balancing  Cost  and  Technical  Performance 


Time  Factors 


Realistic  understanding  of  continually  evolving  customer  Exponential  increase  in  costs 

environments  downstream 


Developing  and  implementing  validated  techniques  Mismatch  in  technical  performance 

to  balance  cost  and  performance  requirements  versus  program  budget 


Availability  of  global  rapidly  emerging  technologies 


Inflexible,  non-scalable  designs 


Impact  of  operational  changes 


System  requirements  obsolete 
before  deployment 


Life  cycle  planning 


O&M  infrastructure 
costs  vs.  service  levels 
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Constructing  the  Program  Performance  Framework 
Using  the  SE  Vee  and  CMMI  Practices 


Guidance.... 


Focus  on  defining  business  goals  and  related  measurements 
entire  period  of  program  performance. 


Plan  and  implement  SE  Vee  model  in  projects  across  the  organization 
sooner  rather  than  later  as  retrofitting  is  difficult. 

Focus  on  measurement  processes  on  forecasting  yearly  costs, 
required  technical  performance  levels  and  program  support  levels. 

Engineer  a  process  performance  framework  using  the  set  of  SE  activities 
represented  in  the  SE  Vee. 

Apply  SE  tools  and  techniques,  such  as  alternative  evaluations, 
performance  simulations,  requirements  definition  and  risk  analysis 
across  the  infrastructure  functions  as  well  as  technical  services  using 
practices  in  the  CMMI. 


About  the  SE  Vee  Model 


Operations 
Concept 

Architecture 

Design 

Development 


Operations 
(including 
D  &  D) 


Deployment 


Verification 


Integration 


Test 


Maturit 


■  The  SE  Vee  Life  Cycle  Model  presented  to  the  Texas  Board  of  Professional  Engineers,  1999, 
by  Arunski,  Martin,  Brown  and  Buede. 

■  The  phases  in  the  Vee  are  traditionally  applied  to  engineering  products  and  services  such  as 
weapons  systems,  communications  networks  and  technical  support. 

■  In  any  program,  phases  in  the  Vee  may  not  be  performed  or  applicable  or  may  exist 
in  numerous  projects  at  different  times. 


Key  infrastructure  functions,  such  as  finance,  contracts,  and  HR  benefit 

from  implementing  the  same  engineering  discipline  and  activities  as  technical  projects. 
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Engineering  of  Process  Performance  Models 


Operations 
Concept 

m 

Architecture  @ 
Design 
Development 


Operations 


“Vee r  Activity 

Operation 

Concept 

Architecture 


Design 


Example  Critical  Support  Functions 

Resources  (space, 
accounting,  BP  systems) 

Business  goals  performance  intervals 

Structure  of  business  performance 
interfaces  (receivables,  quality  measures 
inventory,  growth) 

Performance  constraints  for  cash  flow, 
service  level  performance 


Development 


Increments  to  support  planned  site  expansion 
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Operations 


Engineering  of  Program  Process  Performance  Models 

CMMI  Process  Area 
Categories 

Project  Management 

(Project  Planning,  Project  Monitoring  &  Control, 

Risk  Management,  Integrated  Project  Management, 
Integrated  Teaming,  Integrated  Supplier  Management, 
Quantitative  Project  Management) 


Concept 
Architecture 
Design 

IH 

Development 

Test 

Integration 


Process  Management 

(Organizational  Process  Focus, 

Organizational  Process  Definition, 
Organizational  Training,  Organizational  Process 
Performance,  Organizational  Innovation  and 
Deployment) 


Engineering 

(Requirements  Management, 

Requirements  Development,  Technical  Solution, 
Product  Integration,  Verification,  Validation) 


Verification 

Deployment 

Operations 


Support 

(Configuration  Management, 

Process  &  Product  Quality  Assurance, 

Measurement  &  Analysis,  Causal  Analysis  &  Resolution, 
Decision  Analysis  &  Resolution, 

Organizational  Environment  for  Integration) 


Engineering  of  Support  Function  Framework 


Operations 
Concept 

Architecture 


Design 

Development 


If 


Operations 

Deployment 


Verification 

Integration 


Test 


CMMI  Product 
Suite 


“Vee”  Phase  Example  Key  Support  Functions 


Key  CMMI  PAs 


Operations 

Concept 

Architecture 

Design 

Development 


Resources  (space,  BP  systems,  staffing  levels) 

Business  goals  performance  intervals 

Structure  of  business  performance 
interfaces  (cash  flow,  quality  measures, 
inventory,  growth,  .etc.) 

Performance  constraints  for  cash  flow, 
service  performance,  staffing 


M&A,  PP,  RSKM 
M&A,  RD 
M&A,  TS,  PI 


M&A,  RD,  RM,  TS 


Builds  to  support  planned  market  and  program  M&A,  RD,  PP,  RSKM 

expansion  8 


Engineering  of  Support  Function  Framework  (Continued) 


Operations 

Concept 


Architecture 
Design 
Development 


Operations 

Deployment 


Verification 

Integration 


Test 


“Vee”  Phase 


Examples  Key  Support  Functions 


Key  CMMI  PAs 


Test 

Integration 

Verification 


Finance  test  scenarios  and  databases 

New  interfaces  of  components  (acquisitions) 
for  growth  goals,  finance  and  HR  functions 

Invoicing  and  staffing  processes 


M&A,  VER,  VAL 
TS,  PI 

M&A,  VER,  VAL 


Deployment 


Perfective  and  adaptive  maintenance  of  support  PP,  PMC,  TS 

functions 


Operations 


Forecasting  of  staffing  and  facilities  costs 


PP,  PMC,  QPM,  9 
OPP,  OID 


Overview  of  the  SE  Vee,  CMMI  Process  Areas  and  Business  Goals 


Operations 


Operations 


Concept 

Architecture 

Design 

Development 


Vee 


Deployment 


m  • 


Verification 

Integration 


Test 


Key  Process  Areas 

Measurement  &  Analysis,  Project  Planning,  Project  Monitoring  &  Control, 

Risk  Management,  Quantitative  Project  Management,  Organizational  Innovation  &  Deployment, 
Causal  Analysis  &  Resolution,  Decision  Analysis  &  Resolution... 


. . 


Program 
Business 
Goals  / 


Support 


Technical 


Customer 

Satisfaction 


Invoicing  Latent  Defects  Customer 

Procurement  Satisfaction 

Contracts 
Staffing 


to 


Case  Study  Example  of  Balancing  Cost  and  Technical  Performance 

in  a  Small  Setting 


Key  CMMI  Process  Areas  j 

Measurement  &  Analysis,  Project  Planning,  Project  Monitoring  &  Control,  | 

Risk  Management,  Quantitative  Project  Management,  Organizational  Innovation  &  Deployment; 
Causal  Analysis  &  Resolution,  Decision  Analysis  &  Resolution...  i 


Practices 


Program 

Business 

Goals 


Support 


Technical 


Customer 

Satisfaction 


Invoicing 

Procurement 

Contracts 

Staffing 


Latent  Defects 


Customer 

Satisfaction 


Key 

Measurements 


Process 

Performance 

Intervals 


Catches/Escapes 


Customer 

Satisfaction 
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Case  Study  Example  of  Balancing  Cost  and  Technical  Performance 

in  a  Small  Setting  (Continued) 


Program 

Business 

Goals 

Key 

Measurements 


Support 


Invoicing 

Procurement 

Contracts 

Staffing 


Technical 


Customer 

Satisfaction 


Latent  Defects  Customer 

Satisfaction 


Process 

Performance 

Intervals 


-6.0%  <  Staff  Size  Accuracy  >  9.0%  o  <  Latent  Defects  >  3  4.5  <  Customer  Satisfaction  >  5.0 

-8.1%  <  Invoice  Accuracy  >  6.5% 
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Summary 


Lessons  Learned . 

S  The  phases  in  the  SE  Vee  provide  a  useful  and  applicable 
life  cycle  model  for  engineering  of  a  framework  to  integrate 
management  and  technical  practices  across  a  program. 


s  The  SE  Vee  is  very  adaptable  to  small  settings  and  applies  to  support  services, 
such  as  finance,  contracts  and  HR. 


S  The  practices  in  the  current  version  of  CMMI  Process  Areas  cover  a  large 
percentage  of  the  phases  in  the  Vee. 

S  For  best  results,  focus  on  first  defining  business  goals  and  relevant 
measurements  to  implement  continuous  process  improvement  to  achieve 
a  balance  of  cost  and  technical  performance  via  the  CMMI. 
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Application  of  Risk  Management  in 

a  Net-Centric  EnvironEfirt&nt 


Rebecca  M.  Cowen-Hirsch 

Vice  Component  Acquisition  Executive 
Defense  Information  Systems  Agency 

703-882-2533 

Rebecca.Cowen-Hirsch@disa.mil 


UNCLASSIFIED 


UNCLASSIFIED 


Traditional  Risk  Management 


*  RISK  is  a  measure  of  potential  inability  to  achieve 
overall  program  objectives  within  defined  cost, 
schedule  and  technical  constraints 


•  RISK  MANAGEMENT  is  the  act  of  of  dealing  with  RISK 


SOURCE:  Risk  Management  Guide  for  DOD  Acquisition,  V2.0,  June  2003 


UNCLASSIFIED  2 


UNCLASSIFIED 


The  Times  ...  They  are  Changing 


aoS'a»oidancT.Seek  Net -Centric  Transformation 


Controlled  Environment 


Risk  Avoidance 


Risk  Management 


environmerf 


...An  impossibility  in  era  of  Net 
Centricity  -  risk  must  be  an 
accepted  fact  of  life 

Net  Centric  World  Wide  Web 


fterif 


Old  risks  have  not  disappeared  and  new  risks  abound 
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UNCLASSIFIED 


Net-Centric  Transformation 


*  Old  Information  Systems  World 


New  Net-centric  World 


-  Systems  oriented 


Services  oriented 


-  Specific  design  requirements  process 
(dream  and  develop) 

-  R&D  from  scratch  V 

-  Years  spent  developing  entire  system 

Is- 

-  Tightly  integrated  functionality 

-  Test  against  perfection 


Functionality-based,  “close 
enough”  (see  and  use) 

Greater  use  of  COTS, 
especially  for  enterprise 
services 

More  focused  on 
sustainability  and  scalable 
deployment 

Dynamic  functionality 
through  composability 


-  System-Level  Security 

-  Obsolescence  and  disposal 

-  Prescriptive  Governance 


-  Security  built  in,  with 
balanced  risks 

-  Living  and  adaptable 

-  Collaborative  governance 


Net-Centric  Environment  raises  different  risk  management  challenges 


UNCLASSIFIED 


How  Is  Risk  Management  Conditioned 
by  Net-Centric  Transformation? 

Emphasis  on  the  use  of  COTS 


Use  of  spiral,  incremental  capabilities  development 
strategies 

Dynamic  test  environments 
Warfighter  need  for  “early-to-market”  product  delivery 


Abbreviated  Milestone  development  process 
Immediacy  to  vulnerability  exploitation  via  web 


All  the  above  and  more  increase  pressure  on  risk 
management  mitigation  strategies  and  tactics 
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UNCLASSIFIED 


Risk  Management  Process 


Risk  Assessment 


Risk  Prioritization 


Risk  Mitigation 


How  will  Risk  Management  differ  in  the 
Net -Centric  Environment? 
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UNCLASSIFIED 


Risk  Identification 


Notional  Risk  Framework  for 
Net-Centric  Environment 


Technological  Risk 

-  Standards  maturity 

-  Vendor  products  stability  and  interoperability 

-  Scalability 

-  Security  challenges 

Deployment/Provisioning  Risk 

-  Predicting  results  of  service  composition 

-  Testing  and  certification  of  services 

-  Service  Release  management 

-  Blend  of  managed  services  from  others  and  own-provided  services 

Business  Risk 

-  New  contracting  models  and  incentives  (not  LSI) 

-  Cost  projection  not  focused  on  development/fielding  cost 

-  Cost-recovery  models  for  services 

-  Market  effect  and  agility  to  mature  services  based  on  usage 

-  Do  we  understand  the  marketplace?  Who  is  offering  competing 
services?  How  much  usage  is  “our”  capability  module  getting? 

Organizational  Risk 

-  System  engineering  process  -  adapted  to  services  model 

-  Risk  management  process  -  aligned  with  outcomes 

-  Governance  process 

-  Life  cycle  management 

-  Staffing  /  skills  /  experience  matches 

-  Right  partnerships?  With  industry?  With  other  government 
organizations? 


Capability-Based 
Acquisition 

•Focus  on  outcomes  linked  to 
operational  use 
•Less  constraining  requirements' 
place  more  decision  options  with 
program 


Systems  Engineering 
for  Net-Centricity 

•Concurrent  engineering  of  multiple 
material  solutions/delivery  model 
•Rapid  fielding,  agility,  and  service 
refresh  must  be  enabled  (not 
>nstrained) 


Challenge:  Not  Just  Cost,  Schedule,  Performance  ... 
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Risk  Assessment  for  Net- 
Centric  Environment 


Traditional  approaches  still  apply 

-  Probability  of  occurrence 

-  Severity  of  impact  of  occurrence 

But,  ... 


-  Complexity  or  displacement  may  mask  effective  valuation 

-  “Contract,  buy,  build”  decreases  emphasis  on  development  control 
factors  (schedule  and  cost)  and  increases  emphasis  on  performance 
and  utility 

-  We  must  learn  to  quantify  risk  in  the  face  of  more  degrees  of 
uncertainty 

•  Looking  for  sources  of  lessons  learned 

•  Early  adopters,  pilots,  commercial  lessons  learned,  academic  study? 


Risk  Prioritization  for 
Net-Centric  Environment 


•  Prioritization:  Use  of  a  decision  framework  that  identifies 
and  prioritizes  risks,  such  that 


-  Risks  with  greatest  impact  and  the  greatest  probability  of  occurring 
are  handled  first  High 


*  But ... 

-  Expand  risk  model  elements 

-  View  multi-dimensionally 

-  Devise  new  metrics  for  prioritizing 
risk 

-  Results  may  revise  material 
solutions  and  Acquisition  Strategy 


Challenges:  Optimize  ROI  /  Link  Risks  to  Strategy  / 


View  risks  in  aqqreqate 
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Risk  Mitigation  for 
Net-Centric  Environment 


•  Prioritization  identifies  highest-impact  targets  for 
risk  mitigation 

*  Decision-makers  must  then  lay  out  options  for  risk 
mitigation  and  consider  resource  investment,  before 
choosing  a  course  of  action 

-  Traditional  Risk  Mitigation  options 

•  Risk  avoidance  (traditionally,  our  first  option) 

•  Risk  transfer/distribution  to  another  party,  e.g.  contractor 

•  Reducing  negative  effect  of  risk 

•  Accepting  consequence  of  risk 

-  New  business  models  may  offer  new  options,  or  change  our 
perception  about  acceptable  mix  of  options 


Challenge:  Finding  new  balance  between  risk  mitigation  and 

risk  assumption  «> 


UNCLASSIFIED 

Summary 

•  Dynamics  of  net-centric  environment  put  different  pressures  on 
risk  management 

•  Old  risks  have  not  disappeared  and  new  risks  abound 

•  Emphasis  shift  from  risk  avoidance  to  risk  management 

•  New  dimensions  to  consider  beyond  cost,  schedule  &  performance 

•  Need  to  continually  revisit  key  decisions  to  assure  they  still  apply 

•  Unknowns  still  to  be  investigated,  studied  and  discussed 

•  Challenges  remain  in  understanding  complexities  and  implications 
of  net-centric  and  service  oriented  architectures  ... 


Net-Centric  Risk  Management  process  ... 

Challenges  to  address 


UNCLASSIFIED 


Challenges 


•  Risk  Identification:  Expanding  the  framework 
beyond  Cost,  Schedule,  Performance  ... 

•  Risk  Assessment:  Credible  valuation  is  the 
foundation  -  We  don’t  know  what  we  don’t  know  (but 
we  need  to  learn  quickly!) 

•  Risk  Prioritization:  Still  need  to  optimize  ROI  -  CBA 
demands  that  we  also  link  risks  to  materiel  solutions 
strategy 

•  Risk  Mitigation:  Risk-averse  culture  must  balance 
risk  mitigation  and  risk  assumption 


UNCLASSIFIED  12 


Improving  M&S  Support  to  Acquisition: 

A  Progress  Report  on  Development  of  the 

Acquisition  M&S  Master  Plan 

Fred  Myers,  OUSD  (AT&L)  DS/SE 
Jim  Hollenbach,  Simulation  Strategies,  Inc., 


NDIA  Systems  Engineering  Conference 
26  October  2005 


Introduction 


□  This  presentation  key  aspects  of  the  emergent  DoD 
Acquisition  M&S  Master  Plan 

>  Background 

>  Process 

>  Draft  action  set 

□  Questions  and  comments  are  invited  here  as  time  permits 

□  NDIA  M&S  Committee  meeting  1445-1730  Thursday  to 
answer  remaining  questions  and  discuss  your  change 
recommendations 

>  All  are  welcome  to  attend 


Senior  Leadership  Imperatives 


Under  Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics: 

□  "Provide  a  context  within  which  I  can  make 
decisions  about  individual  programs." 

□  "Achieve  credibility  and  effectiveness  in  the 
acquisition  and  logistics  support  processes." 

□  "Help  drive  good  systems  engineering  practices 
back  into  the  way  we  do  business." 


Response:  Establish  an  SE  Office 

Defense  Systems  Directorate,  OUSD(AT&L) 


An  integrated  structure  to  develop  capability 


M&S  is  a  Necessary  Part  of  Acquisition 


M&S  is  broadly  useful  to  enable  systems  engineering 
throughout  a  system  or  S-o-S  life  cycle 


Recursive 

Processes 


User  Requirements 
&  Concept  of 
Operations 


System 

Requirements  & 
Architecture 


System 

Demonstration  & 
Validation 


System  Integration 
&  Test 


Component  Design  v 


Component 
Integration  &  Test 


Procure,  Fabricate, 
&  Assemble  Parts 


Recursive 

Processes 


Iterative  "V"  process  across  acquisition  phases 


Acquisition  M&S  Working  Group 

Per  AMSWG  Charter,  approved  by  SE  Forum  Feb  2005 

...anchored  in  acquisition  community, 
linked  to  industry  and  M&S 


Industry  /  Academia 

SISO.  OMG.  etc. 

INCOSE 

NDIA 

SE  Division 

SAPD,  SE  DSIG,  etc. 

INCOSE  MDSD  WG 

NDIA 

M&S  Committee 


Products 


DoD  SE  DoD  M&S 


Systems 

Engineering  Forum 


Chair:  Mark  Schaeffer 


EXCIMS 


Acquisition 

M&S 

M&S  Working  Group 

Working  Group 

Chair:  Fred  Myers 

Product 

Product 

Reports  (e.g.,  “M&S  Support 
to  the  New  DoD  Acq.  Process”), 
standards,  papers,  etc. 


Acquisition  M&S 
Master  Plan 


DoD  M&S 
Master  Plan 


Approach 


□  Foster  widely-needed  M&S  capabilities  that  are  beyond  the 
reach  of  individual  programs 

□  Address  M&S  issues  and  actions  necessary  to  enable 
acquisition  of  effective  joint  capabilities  (systems  of  systems) 

□  Not  seek  to  do  the  job  of  program/capability  managers;  rather 
seek  to  empower  them 

>  By  removing  systemic  obstacles  in  their  path 

>  By  identifying  new  options  for  approaching  their  tasks 

>  By  helping  meet  widely-shared  needs 
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A  Decade  of  Studies  on 
M&S  Support  to  Acquisition 


1.  Final  Report  of  the  Acquisition  Task  Force  on  M&S,  1 994 

Sponsor:  DDR&E  (Dr.  Anita  Jones);  Chair:  VADM  T.  Parker,  USN  (Ret.) 

2.  Naval  Research  Advisory  Committee  Report  on  M&S,  1 994 

Sponsor:  ASN(RDA);  Chair:  Dr.  Delores  Etter 

3.  Collaborative  Virtual  Prototyping  Assessment  for  Common  Support 
Aircraft,  1995 

Sponsor:  Naval  Air  Systems  Command;  conducted  by  JHU  APL  and  NSMC 

4.  Collaborative  Virtual  Prototyping  Sector  Study,  1 996 

North  American  Technology  &  Industrial  Base  Organization;  sponsor:  NAVAIR 

5.  Application  of  M&S  to  Acquisition  of  Major  Weapon  Systems,  1 996 

American  Defense  Preparedness  Association;  sponsor:  Navy  Acqn.  Reform  Exec. 

6.  Effectiveness  of  M&S  in  Weapon  System  Acquisition,  1 996 

Sponsor:  DTSE&E  (Dr.  Pat  Sanders);  conducted  by  SAIC  (A.  Patenaude) 

7.  Technology  for  USN  and  USMC,  Vol.  9:  M&S,  1 997 

Naval  Studies  Board,  National  Research  Council;  sponsor:  CNO 

8.  A  Road  Map  for  Simulation  Based  Acquisition,  1 998 

Joint  SBA  Task  Force  (JHU  APL  lead);  sponsor:  Acquisition  Council  of  EXCIMS 
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A  Decade  of  Studies  on 
M&S  Support  to  Acquisition 


9.  M&S  for  Analyzing  Advanced  Combat  Concepts,  1 999 

Defense  Science  Board  Task  Force  (Co-chairs:  L.  Welch,  T.  Gold) 

10.  Advanced  Engineering  Environments,  1999 

National  Research  Council;  sponsor:  NASA 

1 1.  Survey  of  M&S  in  Acquisition,  1 999  and  2002 

Sponsor:  DOT&E/LFT&E;  conducted  by  Flicks  &  Associates  (A.  Hillegas) 

12.  Test  and  Evaluation,  1 999 

Defense  Science  Board  Task  Force  (Chair:  C.  Fields) 

13.  ' SIMTECH  2007 ”  Workshop  Report,  2000 

Military  Operations  Research  Society  (Chair:  S.  Starr) 

14.  M&S  in  Manufacturing  and  Defense  Systems  Acquisition,  2002 

National  Research  Council;  sponsor:  DMSO 

15.  M&S  Support  to  the  New  DoD  Acquisition  Process,  2004 

NDIA  Systems  Engineering  Div.  M&S  Committee;  sponsor:  PD,  USD(AT&L)DS 

16.  Missile  Defense  Phase  III  M&S,  2004 

Defense  Science  Board  Task  Force  (Chair:  W.  Schneider) 
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Assessment  of  Current  Issues/Needs 

□  Cooperative  effort  between  AMSWG  &  NDIA  M&S  Committee 

□  AMSWG  venue: 

>  Air  Force  -  Roe  (Jan  05) 

>  Army  -  Gillis,  Wallace  (Jan  05) 

>  Navy  -  Vaughn  (Feb  05) 

>  Visits  to  NAWC/AD  (ACETEF);  Army  RDECOM;  AFMC  (SIMAF,  ICE) 

□  NDIA  M&S  Committee  venue: 

>  Joint  SIAP  Systems  Engineering  Organization  (Aug  04) 

>  Future  Combat  Systems  (Sep  04) 

>  Missile  Defense  Agency  (Feb  05) 

>  Lockheed  Martin  (Feb  05) 

>  Raytheon  (Apr  05) 

>  Boeing  (Apr  05) 

>  Northrop  Grumman  (Jun  05) 

>  BAE  Systems  (Aug  05) 

□  Affirmed  many  findings  and  recommendations  from  studies  and  provided 
new  inputs  as  well 


Content  of  Acquisition  M&S  Master  Plan 


Forward 

Purpose 

Background 

Vision 

Objectives  (5) 

Actions  (28) 

■  Action 

■  Rationale 
J  ■  Discussion 

■  Lead  &  supporting  organizations 

■  Products 

V  Completion  goal  (year) 


DoD5000J1«PH 


nr 

DoD  5D0O.59-PH 


Acquisition 

Modeling  and  Simulation 
Master  Plan 


Preliminary  Draft 


Date 

Under  secretary  ef  Defense  for 

Acquisition,  Technology,  and  Logistics 


Execution  Management 


Objective  1 


Provide 
necessary 
policy  and 
guidance 


Actions 

1.  M&S 
management 

2.  Model-based 
systems 
engineering  & 
collaborative 
environments 

3.  M&S  in  testing 

4.  M&S  planning 
documentation 

5.  RFP  &  contract 
language 

6.  Security 
certification 


Five  Objectives 

(buckets  for  28  actions) 


Objective  2 


Enhance  the 
technical 
framework 
for  M&S 


Actions 

7.  Data  standards 
framework 

8.  Product 
development 
metamodel 

9.  Commercial 
SE  standards 

10.  Distributed 
simulation 
standards 

11.  DoDAF  utility 

-  DoDAF  2.0 
Acqn  Overlay 

-  Standards  for 
depiction  & 
interchange 

12.  Metadata 
template  for 
reusable 
resources 


Objective  3 


Improve 
model  and 
simulation 
capabilities 


Actions 

13.  Acquisition 
inputs  to  DoD 
M&S  priorities 

14.  Best  practices 
for  model/sim 
development 

15.  Distributed  LVC 
environments 

-  Standards 

-  Sim/lab/range 
compliance 

-  Event  services 

16.  Central  funding 
of  high-priority, 
broadly-needed 
models  &  sims 

-  Prioritized  needs 

-  Pilot  projects 

-  Expansion  as 
warranted 


Objective  4 


Improve 
model  and 
simulation 
use 


Actions 

17.  Help  defining 
M&S  strategy 

18.  M&S  planning 
&  employment 
best  practices 

19.  Foster  reuse 

-  Business  model 

-  Responsibilities 

-  Resource 
discovery 

20.  Info  availability 

-  Scenarios 

-  Systems 

-  Threats 

-  Environment 

21. VV&A 

-  Documentation 

-  Risk-based 

-  Examination 

22.  COTS  SE  tools 

23.  M&S  metrics 


Objective  5 


Shape  the 
workforce 


Actions 

24.  Definition  of 
required  M&S 
competencies 

25.  Harvesting  of 
commercial 
M&S  lessons 

26.  Assemble  Body 
of  Knowledge 
for  Acqn  M&S 

27.  M&S  education 
&  training 

-  DAU,  DAG  & 
on-line  CLMs 

-  Conferences, 
workshops  & 
assist  visits 

28.  MSIAC  utility 
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Objective  1 :  Provide  Necessary  Policy  &  Guidance 


(Preamble)  Need  to  assign  responsibility  for  management  of  the  joint  capability 
areas,  to  include  systems  engineering  and  its  M&S  component 

1 .  Provide  effective,  persistent  DoD-wide  M&S  management  to  address 
cross-cutting  M&S  issues,  coordinate  actions 

Lead:  OUSD(AT&L)/DDRE;  Support:  OUSD(AT&L)/DS,  Components 
Products:  Revised  DoDD  5000.59  (M&S  Management)  with  clearer  responsibilities,  revised 
EXCIMS  membership,  SOP  for  EXCIMS  processes,  a  refocused  DMSO 

Completion  goal:  2006 

2.  Promote  model-based  systems  engineering  (MBSE)  and  M&S-enabled 
collaborative  environments,  at  both  the  program  and  joint  capability  level 

Lead:  OUSD(AT&L)/DS;  Support:  Components 
Products:  Revised  guidance  in  DAG 

Completion  goal:  2007 

3.  Establish  policy  on  appropriate  use  of  M&S  to  plan  tests,  to  complement 
system  live  tests,  and  to  evaluate  joint  capabilities 

Co-leads:  OUSD(AT&L)/DS,  ODOT&E;  Support:  Components 
Products:  Revised  policy  and  guidance  in  DoDI  5000.2  and  DAG 

Completion  goal:  2006 
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Obj.  1 :  Provide  Necessary  Policy  &  Guidance  (cont.) 


4.  Establish  policy  to  require  documented  M&S  planning  at  the  joint  capability 
&  program  levels  as  part  of  the  Systems  Engineering  Plan,  T&E  Strategy 
and  T&E  Master  Plan 

Co-leads:  OUSD(AT&L)/DS,  ODOT&E;  Support:  Components 

Products:  Revised  policy  and  guidance  in  DoDI  5000.2,  DAG,  and  DOT&E  TEMP  Planning 
Guidance 

Completion  goal:  2006 

5.  Establish  guidelines  for  M&S-related  RFP  language  &  contract  provisions 

Lead:  OUSD(AT&L)/DS;  Support:  OUSD(AT&L)/DPAP,  Components 
Products:  Sample  language  in  DoD  publications  (e.g.,  DAG,  SEP  Preparation  Guide, 
Contracting  for  Systems  Engineering  Guidebook)  regarding  M&S  requirements,  data 
rights,  and  the  responsibilities  and  liabilities  of  parties  regarding  sharing  and  reuse 

Completion  goal:  2006 

6.  Publish  practical  guidelines  for  security  certification  of  M&S  activities  falling 
under  multiple  Information  Assurance  Defense  Accreditation  Authorities 

Lead:  OASD(NII);  Support:  OUSD(AT&L)/DS,  NSA 

Products:  Guidelines  published  in  DoD  8500. 2-H,  per  DoDI  8500.2  “Information  Assurance 
Implementation,”  Feb  6,  2003 

Completion  goal:  2007 
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Objective  2:  Enhance  the  Technical  Framework  for  M&S 

7.  Establish  a  framework  for  data  interchange-related  standards 

Lead:  OASD(NII);  Support:  OUSD(AT&L)/DS 
Products:  Revised  guidance  in  Nil  policy  documents 

Completion  goal:  2008 

8.  Develop  a  product  development  information  metamodel  &  associated 
metadata  extensions  to  the  DoD  Discovery  Metadata  Specification 

Lead:  OUSD(AT&L)/DS;  Support:  OASD(NII),  Components 
Products:  Revised  DDMS;  revised  guidance  in  DAG. 

Completion  goal:  2008 

9.  Support  development  of  open  commercial  systems  engineering-related 
standards,  such  as  OMG’s  Systems  Modeling  Language  (SysML)  and 
ISO  Standard  1 0303  AP-233 

Lead:  OUSD(AT&L)/DS;  Support:  DLA,  OUSD(AT&L)/DDRE,  OASD(NII) 

Products:  Published  standards  suitable  for  adoption  by  DoD 

Completion  goal:  2007 

10.  Establish  a  forum  to  clarify  the  characteristics  and  application  of  various 
distributed  simulation  standards  (HLA,  TENA,  DIS,  ALSP,  SI3,  etc.)  and 
examine  opportunities  for  convergence 

Lead:  OUSD(AT&L)/DDRE  Support:  OUSD(AT&L)/TRMC  &  DS,  ODOT&E,  Components 
Products:  (1)  Information  on  strengths  &  weaknesses  of  the  various  standards;  (2) 

agreement  on  policy  and/or  guidance  on  the  use  of  distributed  simulation  standards;  (3) 
a  way  ahead  regarding  distributed  simulation  standards 

Completion  goal:  2007 
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Obj.  2:  Enhance  the  Technical  Framework  for  M&S  (cont.) 


1 1 .  Improve  the  utility  of  the  DoD  Architecture  Framework  (DoDAF)  for 
acquisition 

11-1.  Develop  the  Acquisition  Overlay  (profile)  for  DoDAF  v2.0 
Lead:  OUSD(AT&L)/DS;  Support:  OASD(NII),  Components 
Products:  Acquisition  Overlay  for  DoDAF  v2.0 

Completion  goal:  2006 

1 1  -2.  Support  development  of  open  commercial  standards  for  the 
depiction  and  interchange  of  DoDAF-compliant  architectures 

Lead:  OASD(NII)  Support:  OUSD(AT&L)/DS 

Products:  Published  standards  suitable  for  adoption  by  DoD  in  DoDAF  2.0;  revised 
guidance  in  DAG 

Completion  goal:  2007 

12.  Establish  a  standard  template  of  key  characteristics  (metadata)  to 
describe  reusable  M&S  resources 

Lead:  OUSD(AT&L)/DDRE  Support:  OUSD(AT&L)/DS  &  TRMC,  DOT&E; 
Components 

Products:  Published  standard  template;  usage  guidance  in  DAG 

Completion  goal:  2007 
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Objective  3:  Improve  Model  &  Simulation  Capabilities 


13.  Establish  a  process  to  ensure  acquisition  needs  are  reflected  in  DoD 
M&S  priorities,  including  S&T 

Lead:  OUSD(AT&L)/DDRE;  Support:  OUSD(AT&L)/DS 

Products:  Incorporate  in  M&S  requirements  process  (ref:  DoD  M&S  Master  Plan)  a 
method  to  capture  and  prioritize  those  acquisition  needs. 

Completion  goal:  2007 

14.  Define  and  foster  best  practices  for  efficient  development  and  evolution 
of  credible  M&S  tools,  incorporating  user-defined  requirements,  a 
systems  engineering  approach,  and  appropriate  verification  &  validation 

Lead:  OUSD(AT&L)/DDRE;  Support:  OUSD(AT&L)/DS,  Components 
Products:  Best  practices  DMSO  publication,  available  via  MSIAC,  DTIC,  etc.;  DAG 
guidance  to  use 

Completion  goal:  2008 
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Obj  3:  Improve  Model  &  Simulation  Capabilities  (corn.) 


15.  Enable  readily-available  distributed  live-virtual-constructive  environments, 
leveraging  related  initiatives 

15-1 .  Establish  DoD-wide  standards  for  distributed  environments 

Lead:  OUSD(AT&L)/DDRE;  Support:  OUSD(AT&L)/TRMC  &  DS;  ODOT&E;  Components 
Products:  Published  standard;  DODI  (#  TBD)  policy  to  use 

Completion  goal:  2008 

15-2.  Make  candidate  simulations,  labs  and  ranges  compliant  with  these 
standards 

Lead:  Components;  Support:  OUSD(AT&L)/DS  &  TRMC 

Products:  Toolkit  of  live,  virtual  and  constructive  representations  ready  to  be  employed  in 
distributed  events 

Completion  goal:  2009 

15-3.  Provide  services  to  help  plan  and  conduct  distributed  events 

Lead:  Components;  Support:  OUSD(AT&L)/TRMC  &  DDRE,  DISA 
Products:  Fee-based  technical  services  to  help  users  (e.g.,  PMs,  Capability  Managers, 
OTAs)  plan  and  conduct  distributed  events 

Completion  goal:  2009 
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Obj  3:  Improve  Model  &  Simulation  Capabilities  (corn.) 


16.  Centrally  fund  and  manage  the  development  and  maintenance  of  high- 
priority,  broadly-needed  M&S  tools 

16-1 .  Identify  and  prioritize  broadly-needed  M&S  tools 

Lead:  OUSD(AT&L)/DDRE;  Support:  OUSD(AT&L)/DS;  Components 
Products:  Prioritized  list  of  common  M&S  tool  needs 

Completion  goal:  2007 

16-2.  Conduct  one  or  more  pilot  projects  to  develop  new  M&S  tools  or 
update  existing  ones  to  meet  these  needs 

Lead:  OUSD(AT&L)/DDRE;  Support:  OUSD(AT&L)/DS,  Components 
Products:  Proof  of  concept  for  managing  the  development/evolution  of  M&S  tools  to 
meet  broadly-shared  needs 

Completion  goal:  2009 

16-3.  Expand  the  scope  of  central  M&S  tool  management  as  warranted 
by  pilot  project  results  and  the  list  of  common  M&S  needs 

Lead:  OUSD(AT&L)/DDRE;  Support:  OUSD(AT&L)/DS,  Components 
Products:  Optimal  means  to  meet  common  needs  for  M&S  tools 

Completion  goal:  2011 
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Objective  4:  Improve  Model  &  Simulation  Use 

17.  Provide  potential  acquisition  M&S  users  the  knowledge  needed  to 
formulate  an  effective  M&S  strategy  via  ready  access  to  M&S  expertise  and 
information  about  M&S  capabilities  and  gaps,  reusable  resources,  lessons- 
learned,  etc. 

Lead:  OUSD(AT&L)/DS;  Support:  OUSD(AT&L)/DDRE 

Products:  Revised  guidance  in  DAG;  improved  knowledge  base  in  MSIAC;  assist  visits 
(e.g.,  by  OUSD(AT&L)/DS) 

Completion  goal:  2007 

18.  Define  best  practices  for  disciplined  M&S  planning  &  employment 

>  Rigorous  analysis  of  M&S  requirements  and  alternative  solutions,  selection  of  best 
course 

>  Efficient  configuration  management,  initialization,  execution  and  post-run  analysis 

>  Cautions  against  inappropriate  use;  approaches  to  maximize  cost-effective  reuse  across 
lifecycle 

Lead:  OUSD(AT&L)/DS,  Support:  OUSD(AT&L)/DDRE,  Components 
Product:  Revised  best  practices  guidance  in  DAG  and  MSIAC 

Completion  goal:  2007 


21 


Obj.  4:  Improve  Model  &  Simulation  Use  (cont.) 

19.  Facilitate  the  sharing  of  reusable  resources 

19-1.  Establish  a  DoD-wide  business  model  for  compensating  providers 
of  reusable  M&S  resources  (e.g.,  information,  software,  services) 

Lead:  OUSD(AT&L)/DDRE;  Support:  OUSD(AT&L)/DS,  OUSD(P&R),  OUSD(C)/PA&E, 
Components 

Product:  Documented  business  model;  revised  policy  and/or  guidance  in  DoD  5000  series 
and  DAG 

Completion  goal:  2007 

19-2.  Establish  DoD  policy  and/or  guidance  regarding  responsibilities  to 
share,  protect  and  properly  use  information  and  M&S  tools 

Co-Leads:  OASD(NII)  and  OUSD(AT&L)/DDRE;  Support:  OUSD(AT&L)/DS  &  DPAP, 
OUSD(P&R),  OUSD(C)/PA&E,  Components 

Product:  Revised  policy  and/or  guidance  in  various  issuances  (e.g.,  DoD  5000  series,  DAG, 
contracting  guidance) 

Completion  goal:  2007 

19-3.  Enhance  the  means  (e.g.,  directory  service,  registries,  bulletin 
boards)  to  discover  existence  of  reusable  M&S  resources  and  contact 
information 

Lead:  OUSD(AT&L)/DDRE  Support:  OUSD(AT&L)/DS,  OUSD(P&R),  OUSD(C)/PA&E, 
Components 

Product:  Functional  means,  with  appropriate  resources  and  incentives,  and  a  continuous 
improvement  process 

Completion  goal:  2007 
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Obj.  4:  Improve  Model  &  Simulation  Use  (cont.) 


20.  Define  the  types  of  information  DoD  organizations  shall  make  available  to 
others  with  a  valid  need  to  know  and  the  processes  to  obtain  them  (per 
reuse  business  model) 

20-1 .  Scenario  data 

Lead:  OUSD(AT&L)/DDRE  Support:  OCJCS(J8),  OUSD(C)/PA&E,  DIA,  Components 
Product:  Approved  scenarios  and  process  to  obtain 

Completion  goal:  2007 

20-2.  System-related  data 

Lead:  OUSD(AT&L)/DS;  Support:  Components 

Product:  Authoritative  system  data  (characteristics  and  performance,  interactions, 
interfaces,  logistic  support,  etc.)  and  process  to  obtain 

Completion  goal:  2007 

20-3.  Threat  data 

Lead:  DOD  MSEA  for  Threat  Data;  Support:  OUSD(AT&L)/DDRE  &  DS,  Components 
Product:  Authoritative  threat  data  and  process  to  obtain 

Completion  goal:  2007 

20-4.  Natural  environment  data 

Lead:  DoD  Natural  Environment  MSEAs;  Support:  OUSD(AT&L)/DDRE  &  DS, 
Components 

Product:  Authoritative  natural  environment  data  and  process  to  obtain 

Completion  goal:  2007 
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Obj.  4:  Improve  Model  &  Simulation  Use  (cont.) 


21 .  Foster  cost-effective  VV&A 

21-1.  Require  DoD-wide  standardized  documentation  of  VV&A 

Lead:  OUSD(AT&L)/DDRE;  Support:  OUSD(AT&L)/DS,  Components 
Products:  Revised  policy  in  DODI  5000.2  and  5000.61 ;  revised  guidance  in  DAG 

Completion  goal:  2007 

21-2.  Develop  risk-based  methodology  and  associated  guidelines  for 
VV&A  expenditures 

Lead:  OUSD(AT&L)/DDRE;  Support:  OUSD(AT&L)/DS,  Components 
Products:  Updated  DMSO  VV&A  Best  Practices  documents/web  site;  guidance  in 
DAG 

Completion  goal:  2006 

21-3.  Examine  a  program’s  VV&A  when  M&S  informs  major  acquisition 
decisions  and  unambiguously  state  the  purpose,  key  assumptions  and 
significant  limitations  of  each  model/simulation  when  results  are 
presented. 

Lead:  OUSD(AT&L)/DS  Support:  DoD  Components 

Products:  Guidance  &  training  for  oversight  personnel;  updates  to  DAG  Chaps  4  &  9 

Completion  goal:  2006 
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Obj.  4:  Improve  Model  &  Simulation  Use  (cont.) 


22.  Assess  the  use  of  COTS  systems  engineering  tools  (modeling 
environments)  for  collaborative  architecture  development 

Lead:  OUSD(AT&L)/DS;  Support:  OASD(NII),  Components 
Products:  Revised  guidance  in  DAG;  enhanced  M&S  body  of  knowledge  for 
dissemination 
Completion  goal:  2006 

23.  Define  and  capture  meaningful  metrics  for  M&S  utility  in  acquisition 

Lead:  Navy;  Support:  OUSD(AT&L)/DS,  Components 

Products:  Metric  definitions  in  DAG;  methods  to  capture  and  submit  data  in  DAG; 
data  from  individual  projects  in  MSIAC,  Body  of  Knowledge,  etc. 

Completion  goal:  2007 
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Objective  5:  Shape  the  Workforce 


24.  Define  required  M&S  competencies  for  the  acquisition  workforce 

Co-Leads:  DAU  and  OUSD(AT&L)/DS;  Support:  OUSD(P&R),  OUSD(AT&L)/DDRE, 
OUSD(C)/PA&E,  Components 

Product:  Identified  lead  FIPT;  workforce  qualification  requirements;  management  process 
&  structure 

Completion  goal:  2008 

25.  Harvest  lessons  from  commercial  sector  activities  in  the  use  of  M&S  to 
support  product  development 

Lead:  OUSD(AT&L)/DS;  Support:  OUSD(AT&L)/DDRE,  Components 
Products:  Lessons  collected  at  a  defined  site  (TBD);  annual  update  to  best  practices  in 
DAG  of  lessons  from  industry  that  should  be  considered  by  PMs  in  planning  for  M&S 
Completion  goal:  Recurring;  initial  in  2007 

26.  Assemble  and  evolve  the  M&S  Body  of  Knowledge  (information  set) 
relevant  to  acquisition 

Lead:  OUSD(AT&L)/DDRE;  Support:  OUSD(AT&L)/DS,  Components 

Product:  Information  base  available  to  potential  M&S  users  (e.g.,  PMs,  CMs,  OTAs); 

source  material  for  education  and  training 
Completion  goal:  Recurring;  initial  in  2007 
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Obj.  5:  Shape  the  Workforce  (corn.) 


27.  Drawing  on  the  M&S  Body  of  Knowledge,  educate  and  train  the  workforce 
to  achieve  required  M&S  competencies 

27-1.  Provide  M&S  knowledge  via  an  expanded  set  of  DAU  courses,  the 
Defense  Acquisition  Guide,  and  on-line  CLMs 

Lead:  DAU;  Support:  OUSD(AT&L)/DS  &  DDRE,  Components 

Product:  Expanded  set  of  DAU  courses,  improved  M&S  guidance  in  the  Defense 

Acquisition  Guide,  on  line  Continuous  Learning  Modules;  a  better  educated  workforce 

Completion  goal:  2009 

27-2.  Provide  M&S  knowledge  via  conferences,  workshops,  and  assist 
visits 

Lead:  OUSD(AT&L)/DS;  Support:  DAU,  OUSD(AT&L)/DDRE,  Components 
Product:  Annual  outreach  program;  a  better  educated  and  trained  workforce 
Completion  goal:  Recurring;  initial  in  2006 

28.  Improve  the  knowledge  and  expertise  available  through  the  MSIAC  to 
make  it  of  greater  utility  to  the  acquisition  community 

Lead:  OUSD(AT&L)/DDRE;  Support:  OUSD(AT&L)/DS,  OUSD(P&R),  OUSD(C)/PA&E, 
Components 

Product:  Plan  of  action  with  coordinated  MSIAC  CONOPS  &  staffing  requirement;  list  of 
knowledge  shortfalls  that  MSIAC  will  take  on;  success  criteria  &  process  to  bring  MSIAC  up 
to  criteria 

Completion  goal:  2008 


27 


Next  Steps 


□  Broadly  vet  actions  (DoD,  Industry) 

>  Fine-tune  actions,  lead  &  support,  products,  etc. 

□  Finalize  Acquisition  M&S  Master  Plan 

>  Acqn  M&S  Working  Group  consensus 

>  SE  Forum  approval  (Dec  2005) 

>  Informal  DoD  coordination 

>  Formal  coordination  as  a  DoD  issuance? 

>  USD(AT&L)  approve  plan 

□  Implement  plan,  monitor  action  completion 

□  Assess  impact  (metrics) 
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Discussion 

□  Questions  and  comments  are  invited  here  as  time  permits 

□  NDIA  M&S  Committee  meeting  1445-1730  Thursday  to 
answer  remaining  questions  and  discuss  your  change 
recommendations 

>  All  are  welcome  to  attend 
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OSD(ATL)/DTRMC 

Cianciolo,  Lawrence 

DCMA 

Clark,  Doug  (ctr)  * 

DMSO 

Digman,  Emmanuel 

NS  A 

Duesterhaus,  Dave 

DOT&E 

Eadie,  J  Marc  * 

NSWCPC(SOCOM  AE) 

Elliott,  Steve 

OUSD(AT&L)  DS/MW 

Espinosa,  William 

OPNAVN091 

Falkey,  Mark  * 

JNIC/SY  COLEMAN 

Flow,  Rob  * 

Feinberg,  Jerry  (ctr) 

OASD  (PA&E)  CAIG 
MSIAC 

Friedenthal,  Sanford 

LMC 

Furness,  Zach  (ctr) 

MITRE 

Gill,  Jim  * 

DDFP/JTAMDO/CSC 

Gillis,  John  (ctr)* 

ASA(ALT)  RDA  M&S 

Glasow,  Jerry 

DMSO 

Goerger,  Niki  * 

USMA 

Goode,  Francine  * 

NS  A 

Halayko,  Bob 

Hardy,  Dwayne  * 

OUSD(P&R) 

OUSD(AT&L)DS 

Hollenbach,  Jim  (ctr) 

OUSD(AT&L)  DS/SE 

Hutchings,  Charles 

NAVSEA 

Johnston,  Jerald 

JHU/APL 

Keener,  Jim 

IWS  1B%  (NSWCDD) 

Kelly,  Doug 

DIA 

Kline,  Tom 

PEO  IWS 

Lackner,  Bob 

PEO  IWS  IC/  ANTEON 

Long,  James 

Vitech  Corp 

Long,  Vicki 

HQAFMC/XRA 

Mackoy,  Rebecca 

TRADOC 

Manthei,  Jerry 

CNO  N091/N912 

Mathis,  Thomas 

RDECOM/SOSI  Camber 

Matzner,  John 

Army  G-3/5/7/Alion 

McDonnell,  Joe  (ctr) 

RDECOM 

Merrill,  Bruce 

NS  A 

Mitchell,  Jeff 

JHU/APL 

Montoya,  Matt 

JHU/APL 

Myers,  Fred  * 

OUSD(ATL)DS 

Nunez,  Patrick 

RDECOM  TARDEC 

Parmele,  Truman 

OASD/NII 

Pikul,  Ronald 

ASN  RDA 

Pittenger,  Bill  * 

DISA/MITRE 

Placanica,  John  * 

NGA 

Preissman,  Dan 

NAVAIR 

Prill,  Mark  * 

NASA/ESMD 

Prosnik,  George  * 

DAU 

Roe,  James  W 

AF  AFMC/XRA 

Sas,  Robert 

NCEE 

Schmidt,  Karl 

NGA/LMC 

Shah,  Nehal  * 

ASN(RDA)  CHENG 

Solo  sky,  Tom  * 

DCMA 

Suter,  Richard 

OUSD  (AT&L)  (T&E) 

Swift,  Lloyd  ctr 

ASN(RDA)  CHENG 

Tillery,  Gordon  (ctr) 

OUSD(AT&L)  DS/SE 

Tirres,  Carlos  * 

DTRMC 

Truelove,  Michael(ctr) 

OUSD  (AT&L)  DS/SE 

Vaughn,  Barbara 

ASN  RDA  CHENG 

Walker,  Oral 

PMUA  MSMO 

Wallace  Jim 

Army  G37/BCSE 

Walsh,  John  * 

OUSD(P&R) 

Wright,  Susan 

OSD  DOT&E 

Znachko,  Carrall 

SAF/AQIZ 

*  Indicates  SE  Forum  Representatives  that  are  AMSWG  Voting  Members 
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Top-Down  Derivation/Cross-Check 


Desired  Acquisition 
Environment  per  CJCSI 
3170  and  DoDD  5000 


ft, 


Needed  Systems 
Engineering  Capabilities 


Characteristics  annotated  as  AE1,  AE2,  ...  AEn 


^  Annotated  as  SE1,  SE2, ...  SEn 


/> 


T\ 


M&S  Processes 
for  Better  SE 


\  ft, 


Annotated  as  MSI,  MS2, ...  MSn 


Gaps  in  Enabling  M&S 
Processes 


Annotated  as 
G1 ,  G2, ...  Gn 


\ 

Needed  Actions 

V 

/ 

Annotated  as 
A1,  A2,...An 
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Desired  Acquisition  Environment: 

Key  CJSCI  31 70.01  E  Policies 

AE1 

■  Jojnt  conceots-centric  capabilities  identification  process  to  allow  joint 
forces  to  meet  the  full  range  of  military  operations  and  challenges. . . 

■  Assess  existing  ana  proposed  capabilities  in  light  of  their  contribution 
to  future  joint  allied  and  coalition  operations. . . .  Produce  capability 
proposals  that  consider  the  full  range  of,  DOTMLPF_  solutions  in  order 
to  advance  joint  warfighting  in  a  unilateral  and  multinational  context. 

■  New  solution  sets. . .  crafted  to  deliver  technologically  sound,  testable. 
AE4  sustainable  ancj  affordable  increments  of  militarily  useful  capability. 

AE5 

■  The  FoS  and_  SoS  solutions  may  also  require  systems  delivered  by 
rmjjtipje  sponsors/rnaterjej  developers. AE6 

■  The  process  to  identify  capability  gaps  and  potential  solutions  must  be 
supported  by  a  robust  analytical  process  AE7 

■  JCIDS  implements  a  capabilities-based approach  that. ..requires  a 
^collaborative  process  that  utilizesjoint  concepts  and  integrated 
ae9  architectures  to  identify  prioritized  capability  gaps  and  integrated 

DOTMLPF  and  policy  approaches  to  resolve  those  gaps 

AE11 
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Desired  Acquisition  Environment: 

DoDD  5000.1  Acquisition  Policies 

AE12 

“The  primary  objective  of  Defense  acquisition  is  to  acquire  quality  products  that 
satisfy  user  needs  with  measurable  improvements  to  mission  capability  and 
operational  support,  in  a  timely  manner,  and  at  a  fair  and  reasonable  price.  ” 

AE13  AE14  AE15 

Governing  policies: 

>  Flexibility,  Responsiveness  (time-phased  capabilities,  evolutionary 
acquisition).  Innovation,  Discipline.  Streamlined  Effective  Management 

AE18  AE19 

>  Armaments  Cooperation;  Collaboration:  Competition;  Cost  and 
Affordability:  Cost  Realism:  Cost  Sharing;  Financial  Management; 
Independent  OTAs;  Information  Assurance;  Integration  Supepjggty; 

AE20  Integrated  T&E;  Intelligence  Support;  Interoperability;  Kno wleaoe-Based 
Acquisition:  Legal  Compliance;  Performance-Based  Acquisition; 

AE23  Performance-Based  Logistics:  Products  Services  and  Technologies  [seek 
most  cost-effective  solution  over  the  system's  Ijfe  cycle],  Professional 
Workforce,  Program  Information  [complete,  current,  tailored];  Program 
Stability;  R&D  Protection;  Safety;  Small  Business  Participation;  Software 
Intensive  Systems;  Streamlined  Organizations;  Systems  Engineering: 
AE26Technoloov  Development  and  Transition:  Total  Systems  Approach  ae27 


>  Oct  04  policy  memo:  Technical  reviews . . .  shall  be  event-driven  ae28 
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Necessary  Systems  Engineering  Capabilities 

(which  M&S  can  affect;  derived  from  Desired  Acquisition  Environment) 

SE1.  Early,  continuing  systems  engineering  from  an  SoS/FoS  capabilities 
perspective;  seamless  transition  from  JCIDS  to  acquisition 

(AE1  -3, 5, 9-1 1 ,1 6,20,21 ,25,27) 

SE2.  Lifecycle-wide  exploration  of  the  maximum  available  trade  space, 
including  time-phased  requirements  and  technology  insertion 

(AE1  -5,7,1 0,1 1 ,1 3,1 6,1 9,23-27) 

SE3.  Collaboration  among  all  stake  holders  (multiple  gov’t  and  contractor 
organizations)  for  key  enterprise-level  SE  decisions  (AE6-8, 10,18,22, 25, 27) 

SE4.  Rapid  assessment  of  concept/design  alternatives  <ae2, 4, 7, 10, 14,16,19,25,28) 

SE5.  Comprehensive,  accurate,  event-based  assessment  of  technical 
baselines;  avoidance  of  costly  fixes  for  problems  discovered  late 

( AE2-4,7,9,1 0,1 2-1 7,1 9,20,22,24-26,28) 

SE6.  Focused,  effective  &  efficient  testing;  including  at  the  capability  level 

(AE1 ,2, 4, 5, 9-1 1 ,13,15,19-22,25) 

SE7.  Appropriate  reuse  of  all  resources  -  information,  software  tools, 
expertise,  facilities,  ranges,  etc.  -  across  programs  &  organizations 

(AE4,1 4,1 5,1 9,24,25) 
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(Jl)M&S  Processes  for  Better  Systems  Engineering 

(derived  from  Needed  Systems  Engineering  Capabilities) 

MSI.  Use  of  a  model-based  systems  engineering  approach  (sei,2,4,5) 

(Emerging  concept  under  INCOSE,  OMG,  etc.;  growing  suite  of  COTS  SE  tools) 

>  SE  tool  modeling  environments  to  analyze  requirements,  develop 
architecture,  and  specify  constraints;  views  linked  to,  and  generated 
from,  an  underlying  database 

>  Embedded  simulation  to  verify  the  architecture  and  assess  its  merits 

>  Automated  generation  of  documents/reports 


Capability 
System 

Engineering  | 


Coordinate/support  Development  and  Engineering  Changes 


MS2.  Establishing  M&S-enabled  collaborative  engineering  environments 

(SE1,2,3,4,5,6) 

>  Shared,  authoritative  information 

>  Interoperable  modeling  environ¬ 
ments  for  architecting/design 

■  Models  as  communication  means 

>  Models  &  simulations  to  assess 

■  Option  to  immerse  warfighters,  etc. 

>  Distributed  live-virtual-constructive 
environments  for  integration,  verification,  and  test 


Program 

System 

Engineering 


x 


System 

T&E 


MS3.  Model-Test-Fix-Model  process  across  the  life-cycle  (SE4,s,6) 

>  Better  test  planning,  more  effective  tests 

>  Increased  M&S  validity;  credible  surrogates;  reuse  savings 
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M&S  Processes  for  Better  Systems  Engineering 

MS4.  Harnessing  M&S  knowledge  to  formulate  an  effective  M&S  strategy 

(SE2,3,4,5,7) 

>  Ready  access  to  M&S  expertise  and  information  about  capabilities 
and  gaps  (M&S  holes),  reusable  resources,  lessons-learned,  etc. 

MS5.  Disciplined  M&S  planning  &  employment  <se2,4,5,7) 

>  Rigorous  analysis  of  M&S  requirements,  alternatives,  best  course 

>  Efficient  configuration/initialization,  execution  and  post-run  analysis 

>  Avoid  inappropriate  use;  maximize  cost-effective  reuse  across  lifecycle 

MS6.  Efficient  development/evolution  of  credible  M&S  tools  (SE2,3,5,7) 

>  A  systems  engineering  approach  with  appropriate  V&V 

MS7.  Access/sharing  of  authoritative,  understandable  data  needed  for  M&S 
representations  <se2,3,4,5,7) 

>  Reducing  a  major  time  and  cost  burden  that  inhibits  M&S  use 

MS8.  Inspection  of  M&S  used  to  inform  acquisition  decisions  <se2,s,7) 

>  Examine  capabilities  and  limitations  (VV&A)  of  M&S 

>  During  lead-up  to  program/technical  reviews,  OTRRs,  DABs,  etc. 
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D2 


Gaps 


1.  Management 


Gl.  Robust  but  confused  landscape  of  M&S  activities;  no  clearly 
designated  leadership  or  effective  coordinating  mechanism  (msi-8) 

>  Current  EXCIMS  ineffective;  little  coordination  for  capabilities/SoS/FoS 


G2.  Inadequate  constancy  of  purpose  because  time  to  fix  problems  » tour 
length;  “DoD  has  an  attention  deficit  disorder”  <ms2-7) 

G3.  Gov’t  acquisition  guidelines  don’t  promote  M&S  use  or  reuse  (msi-6) 

G4.  No  DoD  requirement  for  formal  M&S  planning  to  support  acquisition 
(other  than  T&E)  <msi-5) 

G5.  No  contractual  guidelines  regarding  M&S  and  the  data  it  needs  <msi-8) 


G6.  Gov’t  typically  doesn’t  give  contractors  meaningful  M&S  guidance 

(MSI  ,2,6,8) 


G7.  Most  DoD  M&S  takes  a  project,  vice  an  enterprise,  approach  <ms2,3,6,7) 
G8.  No  consensus  on  value  of  integrated  architectures,  nor  responsibility 

for  (MSI ,2) 


G9.  Managing  distributed  collaboration  is  very  hard  (msi-8) 


G10.  Public  law  precludes  OT  based  solely  on  M&S,  but  no  clear  guidance 
on  use  for  SoS/FoS  T&E  <ms2,3,5,6,8) 
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D2 


Gaps 


2.  Architecture/standards/technical  framework 

Gil.  No  standard  modeling  notation  (like  UML  v2.0)  for  capturing  full  range 
of  information  critical  to  system  engineering  (e.g.,  structure,  behavior, 
requirements  hierarchy/traceability,  test  cases,  verification  results)  (msi ,2,6,7) 

G12.  No  standard  for  interchanging  systems  engineering  information  (same 
examples  as  above)  (msi  ,2,6,7) 

G13.  No  conceptual  framework  (like  Open  System  Interconnect  protocol  stack) 
for  data  interchange  (msi,2,3,6,7) 

G14.  Lack  of  agreement  on  a  common  distributed  simulation  standard 
increases  complexity  and  cost,  limits  simulation  interoperability  <ms2,5,6) 

G15.  DoDAF  vl  .0  is  difficult  to  use  for  architecting  due  to  lack  of  data- 
centricity  and  executability;  some  products  of  marginal  value<Msi,2,6,7) 

G16.  Use  of  DoD-unique  standards  limits  their  user  base,  quality,  COTS  tool 
support,  and  opportunities  for  reuse  (msi ,2,5,6) 
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D2 


Gaps 

3.  Model/simulation  capabilities  &  use 

G17.  Many  M&S  tool  gaps  and  deficiencies  <msi,2,3,5,7) 

>  What’s  modeled  (e.g.,  urban  warfare,  comm  networks,  threats,  system  sustainment) 

>  Fidelity,  granularity,  interoperability 

>  Only  limited  consensus  on  common  models  to  be  used  across  a  domain 

G18.  No  good  way  to  develop  and  maintain  widely-needed  M&S  tools  that  cut 
across  programs  (mss,6) 

>  Not  incorporating  mods  by  other  organizations  into  “street  version,”  etc. 

G19.  M&S  developers,  not  M&S  users,  tend  to  drive  M&S  development  <ms6) 

G20.  In  general,  architecture  development  (modeling)  is  lagging,  not 

collaborative,  and  not  exploiting  COTS  SE  tools  (modeling  environments) 

(MSI  ,2) 

G21.  No  readily-available  distributed  M&S  infrastructure  (e.g.,  JDEP)  <ms2,5) 

G22.  Hard  to  get  security  certification  for  multi-organization  (company/ 
Service)  distributed  simulation  <ms2,3,5,6) 

G23.  Hard  to  get  approval  and  security  certification  for  M&S  involving 
multiple  compartmented  programs  (SAPs)  <ms2,3,5,6,7) 
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D2 


Gaps 


4.  Trustworthiness/VV&A 

G24.  Post-development  model  validation  expensive  and  slow  <ms2,3,5,8) 

G25.  VV&A  often  weak  or  non-existent;  documentation  inconsistent 

(MS2,3,5,8) 

>  Plans  to  use  M&S  to  avoid  testing  costs  often  rejected  due  to  poor/no 
validation 

G26.  VV&A  usually  not  enforced  and  also  not  examined  during  program 
reviews  (MS2,3,5,6,8) 

G27.  Models  and  sims  often  not  updated  to  reflect  empirical  evidence 
(e.g.,  test  results)  <ms2,3,5,8) 
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D2 


Gaps 

5.  Sharinq/reuse  and  protection  of  tools  &  information 

G28.  Little  reuse;  only  7%  of  models  &  sims  used  on  >1  program  (MS2,5,6) 

G29.  Concurrent  engineering  requires  an  integrated  process,  data  sharing 
and  a  coherent  tool  set,  but  <20%  of  programs  have  such  a  collaborative 
environment  <ms2,7) 

G30.  Hard  to  discover  reusable  resources  (software,  info,  services)  (MS2,4,5,7) 

>  M&S  repositories  are  not  integrated,  lack  an  effective  search 
capability,  and  are  mostly  empty 

>  MS  I  AC  knowledge/expertise  is  lacking 

G31.  Insufficient  info  (metadata)  to  evaluate  data/reuse  candidates  (ms2,4,5,7) 

G32.  Hard  to  obtain  reusable  resources  <ms2,4,5,7) 

>  Industry  to  gov’t:  To  protect  proprietary  info  &  competitive  advantage 

>  Gov’t  to  industry:  Contractual  liabilities  associated  with  GFE/GFI 

>  Gov’t  to  gov’t:  Concerns  about  misuse;  cost  to  deliver  and  guide 

G33.  No  incentives  to  encourage  reuse  <ms2,3,5,6) 

>  Negative  incentives  include  cost  to  make  reusable,  workload 
assisting  users,  vulnerability  to  criticism 

[plus  approval  and  security  certification  gaps  22  &  23  listed  under  M&S  use] 
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D2 


Gaps 


6.  Research/S&T/tech  base 

G34.  Conceptual  foundation  of  M&S  weak  (mss,6) 

>  E.g.,  theoretical  understanding  of  modern  warfare,  human 
behavior,  relating  M&S  at  different  granularities,  dealing  with 
uncertainty,  agent-based  modeling  and  generative  analysis 

G35.  Little  acquisition  community  input  to  DoD  S&T  management 
regarding  needed  M&S-related  research  (ms2,s,6) 

7.  Business  model,  metrics  &  ROI.  funding  and  incentives 

G36.  No  business  model  for  how  M&S  capabilities  should  be  developed, 
used  and  maintained  <msi-8) 

G37.  Metrics  are  critical  to  keep  interest  and  funding  up,  but  metrics 
regarding  M&S  use  and  cost-effectiveness  are  inadequate  <msi-8) 

>  M&S  funding  difficult  to  identify;  most  embedded  within  other  PEs 

G38.  Too  little  funding  (MS2-7) 
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D2 


Gaps 


8.  Workforce  Shaping 

G39.  Body  of  knowledge  for  M&S  support  to  acquisition  is  deficient,  not 
managed  <msi,2,4-6,8) 

G40.  Acqn  community  managers  and  staffs  mostly  uninformed  about 
M&S  capabilities  and  limitations  <msi-8> 

>  Weak  acquisition  personnel  understanding  of  commercial  M&S 
activities  (“We  don’t  get  out  enough”) 

>  Not  enough  M&S  experts  (no  career  path  [except  Army],  no 
formal  education  or  training) 

G41.  M&S  developers  lack  understanding  of  modeling  best  practices, 
abstraction  techniques,  context  dependencies,  etc.  <ms3,6) 

G42.  M&S  users  often  not  adequately  trained  <msi ,2, 4, 5, 8) 

G43.  Insufficient  M&S  education  options  <ms2,4,5,6,8) 
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Improving  M&S  Support  to  Acquisition: 

A  Progress  Report  on  Development  of  the 

Acquisition  M&S  Master  Plan 

Fred  Myers,  OUSD  (AT&L)  DS/SE 
Jim  Hollenbach,  Simulation  Strategies,  Inc., 


NDIA  Systems  Engineering  Conference 
26  October  2005 


Introduction 


□  This  presentation  key  aspects  of  the  emergent  DoD 
Acquisition  M&S  Master  Plan 

>  Background 

>  Process 

>  Draft  action  set 

□  Questions  and  comments  are  invited  here  as  time  permits 

□  NDIA  M&S  Committee  meeting  1445-1730  Thursday  to 
answer  remaining  questions  and  discuss  your  change 
recommendations 

>  All  are  welcome  to  attend 


Senior  Leadership  Imperatives 


Under  Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics: 

□  "Provide  a  context  within  which  I  can  make 
decisions  about  individual  programs." 

□  "Achieve  credibility  and  effectiveness  in  the 
acquisition  and  logistics  support  processes." 

□  "Help  drive  good  systems  engineering  practices 
back  into  the  way  we  do  business." 


Response:  Establish  an  SE  Office 

Defense  Systems  Directorate,  OUSD(AT&L) 


An  integrated  structure  to  develop  capability 


M&S  is  a  Necessary  Part  of  Acquisition 


M&S  is  broadly  useful  to  enable  systems  engineering 
throughout  a  system  or  S-o-S  life  cycle 


Recursive 

Processes 


User  Requirements 
&  Concept  of 
Operations 


System 

Requirements  & 
Architecture 


System 

Demonstration  & 
Validation 


System  Integration 
&  Test 


Component  Design  v 


Component 
Integration  &  Test 


Procure,  Fabricate, 
&  Assemble  Parts 


Recursive 

Processes 


Iterative  "V"  process  across  acquisition  phases 


Acquisition  M&S  Working  Group 

Per  AMSWG  Charter,  approved  by  SE  Forum  Feb  2005 

...anchored  in  acquisition  community, 
linked  to  industry  and  M&S 


Industry  /  Academia 

SISO.  OMG.  etc. 

INCOSE 

NDIA 

SE  Division 

SAPD,  SE  DSIG,  etc. 

INCOSE  MDSD  WG 

NDIA 

M&S  Committee 


Products 


DoD  SE  DoD  M&S 


Systems 

Engineering  Forum 


Chair:  Mark  Schaeffer 


EXCIMS 


Acquisition 

M&S 

M&S  Working  Group 

Working  Group 

Chair:  Fred  Myers 

Product 

Product 

Reports  (e.g.,  “M&S  Support 
to  the  New  DoD  Acq.  Process”), 
standards,  papers,  etc. 


Acquisition  M&S 
Master  Plan 


DoD  M&S 
Master  Plan 


Approach 


□  Foster  widely-needed  M&S  capabilities  that  are  beyond  the 
reach  of  individual  programs 

□  Address  M&S  issues  and  actions  necessary  to  enable 
acquisition  of  effective  joint  capabilities  (systems  of  systems) 

□  Not  seek  to  do  the  job  of  program/capability  managers;  rather 
seek  to  empower  them 

>  By  removing  systemic  obstacles  in  their  path 

>  By  identifying  new  options  for  approaching  their  tasks 

>  By  helping  meet  widely-shared  needs 
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A  Decade  of  Studies  on 
M&S  Support  to  Acquisition 


1.  Final  Report  of  the  Acquisition  Task  Force  on  M&S,  1 994 

Sponsor:  DDR&E  (Dr.  Anita  Jones);  Chair:  VADM  T.  Parker,  USN  (Ret.) 

2.  Naval  Research  Advisory  Committee  Report  on  M&S,  1 994 

Sponsor:  ASN(RDA);  Chair:  Dr.  Delores  Etter 

3.  Collaborative  Virtual  Prototyping  Assessment  for  Common  Support 
Aircraft,  1995 

Sponsor:  Naval  Air  Systems  Command;  conducted  by  JHU  APL  and  NSMC 

4.  Collaborative  Virtual  Prototyping  Sector  Study,  1 996 

North  American  Technology  &  Industrial  Base  Organization;  sponsor:  NAVAIR 

5.  Application  of  M&S  to  Acquisition  of  Major  Weapon  Systems,  1 996 

American  Defense  Preparedness  Association;  sponsor:  Navy  Acqn.  Reform  Exec. 

6.  Effectiveness  of  M&S  in  Weapon  System  Acquisition,  1 996 

Sponsor:  DTSE&E  (Dr.  Pat  Sanders);  conducted  by  SAIC  (A.  Patenaude) 

7.  Technology  for  USN  and  USMC,  Vol.  9:  M&S,  1 997 

Naval  Studies  Board,  National  Research  Council;  sponsor:  CNO 

8.  A  Road  Map  for  Simulation  Based  Acquisition,  1 998 

Joint  SBA  Task  Force  (JHU  APL  lead);  sponsor:  Acquisition  Council  of  EXCIMS 
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A  Decade  of  Studies  on 
M&S  Support  to  Acquisition 


9.  M&S  for  Analyzing  Advanced  Combat  Concepts,  1 999 

Defense  Science  Board  Task  Force  (Co-chairs:  L.  Welch,  T.  Gold) 

10.  Advanced  Engineering  Environments,  1999 

National  Research  Council;  sponsor:  NASA 

1 1.  Survey  of  M&S  in  Acquisition,  1 999  and  2002 

Sponsor:  DOT&E/LFT&E;  conducted  by  Flicks  &  Associates  (A.  Hillegas) 

12.  Test  and  Evaluation,  1 999 

Defense  Science  Board  Task  Force  (Chair:  C.  Fields) 

13.  ' SIMTECH  2007 ”  Workshop  Report,  2000 

Military  Operations  Research  Society  (Chair:  S.  Starr) 

14.  M&S  in  Manufacturing  and  Defense  Systems  Acquisition,  2002 

National  Research  Council;  sponsor:  DMSO 

15.  M&S  Support  to  the  New  DoD  Acquisition  Process,  2004 

NDIA  Systems  Engineering  Div.  M&S  Committee;  sponsor:  PD,  USD(AT&L)DS 

16.  Missile  Defense  Phase  III  M&S,  2004 

Defense  Science  Board  Task  Force  (Chair:  W.  Schneider) 
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Assessment  of  Current  Issues/Needs 

□  Cooperative  effort  between  AMSWG  &  NDIA  M&S  Committee 

□  AMSWG  venue: 

>  Air  Force  -  Roe  (Jan  05) 

>  Army  -  Gillis,  Wallace  (Jan  05) 

>  Navy  -  Vaughn  (Feb  05) 

>  Visits  to  NAWC/AD  (ACETEF);  Army  RDECOM;  AFMC  (SIMAF,  ICE) 

□  NDIA  M&S  Committee  venue: 

>  Joint  SIAP  Systems  Engineering  Organization  (Aug  04) 

>  Future  Combat  Systems  (Sep  04) 

>  Missile  Defense  Agency  (Feb  05) 

>  Lockheed  Martin  (Feb  05) 

>  Raytheon  (Apr  05) 

>  Boeing  (Apr  05) 

>  Northrop  Grumman  (Jun  05) 

>  BAE  Systems  (Aug  05) 

□  Affirmed  many  findings  and  recommendations  from  studies  and  provided 
new  inputs  as  well 


Content  of  Acquisition  M&S  Master  Plan 


Forward 

Purpose 

Background 

Vision 

Objectives  (5) 

Actions  (28) 

■  Action 

■  Rationale 
J  ■  Discussion 

■  Lead  &  supporting  organizations 

■  Products 

V  Completion  goal  (year) 


DoD5000J1«PH 


nr 

DoD  5D0O.59-PH 


Acquisition 

Modeling  and  Simulation 
Master  Plan 


Preliminary  Draft 


Date 

Under  secretary  ef  Defense  for 

Acquisition,  Technology,  and  Logistics 


Execution  Management 


Objective  1 


Provide 
necessary 
policy  and 
guidance 


Actions 

1.  M&S 
management 

2.  Model-based 
systems 
engineering  & 
collaborative 
environments 

3.  M&S  in  testing 

4.  M&S  planning 
documentation 

5.  RFP  &  contract 
language 

6.  Security 
certification 


Five  Objectives 

(buckets  for  28  actions) 


Objective  2 


Enhance  the 
technical 
framework 
for  M&S 


Actions 

7.  Data  standards 
framework 

8.  Product 
development 
metamodel 

9.  Commercial 
SE  standards 

10.  Distributed 
simulation 
standards 

11.  DoDAF  utility 

-  DoDAF  2.0 
Acqn  Overlay 

-  Standards  for 
depiction  & 
interchange 

12.  Metadata 
template  for 
reusable 
resources 


Objective  3 


Improve 
model  and 
simulation 
capabilities 


Actions 

13.  Acquisition 
inputs  to  DoD 
M&S  priorities 

14.  Best  practices 
for  model/sim 
development 

15.  Distributed  LVC 
environments 

-  Standards 

-  Sim/lab/range 
compliance 

-  Event  services 

16.  Central  funding 
of  high-priority, 
broadly-needed 
models  &  sims 

-  Prioritized  needs 

-  Pilot  projects 

-  Expansion  as 
warranted 


Objective  4 


Improve 
model  and 
simulation 
use 


Actions 

17.  Help  defining 
M&S  strategy 

18.  M&S  planning 
&  employment 
best  practices 

19.  Foster  reuse 

-  Business  model 

-  Responsibilities 

-  Resource 
discovery 

20.  Info  availability 

-  Scenarios 

-  Systems 

-  Threats 

-  Environment 

21. VV&A 

-  Documentation 

-  Risk-based 

-  Examination 

22.  COTS  SE  tools 

23.  M&S  metrics 


Objective  5 


Shape  the 
workforce 


Actions 

24.  Definition  of 
required  M&S 
competencies 

25.  Harvesting  of 
commercial 
M&S  lessons 

26.  Assemble  Body 
of  Knowledge 
for  Acqn  M&S 

27.  M&S  education 
&  training 

-  DAU,  DAG  & 
on-line  CLMs 

-  Conferences, 
workshops  & 
assist  visits 

28.  MSIAC  utility 
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Objective  1 :  Provide  Necessary  Policy  &  Guidance 


(Preamble)  Need  to  assign  responsibility  for  management  of  the  joint  capability 
areas,  to  include  systems  engineering  and  its  M&S  component 

1 .  Provide  effective,  persistent  DoD-wide  M&S  management  to  address 
cross-cutting  M&S  issues,  coordinate  actions 

Lead:  OUSD(AT&L)/DDRE;  Support:  OUSD(AT&L)/DS,  Components 
Products:  Revised  DoDD  5000.59  (M&S  Management)  with  clearer  responsibilities,  revised 
EXCIMS  membership,  SOP  for  EXCIMS  processes,  a  refocused  DMSO 

Completion  goal:  2006 

2.  Promote  model-based  systems  engineering  (MBSE)  and  M&S-enabled 
collaborative  environments,  at  both  the  program  and  joint  capability  level 

Lead:  OUSD(AT&L)/DS;  Support:  Components 
Products:  Revised  guidance  in  DAG 

Completion  goal:  2007 

3.  Establish  policy  on  appropriate  use  of  M&S  to  plan  tests,  to  complement 
system  live  tests,  and  to  evaluate  joint  capabilities 

Co-leads:  OUSD(AT&L)/DS,  ODOT&E;  Support:  Components 
Products:  Revised  policy  and  guidance  in  DoDI  5000.2  and  DAG 

Completion  goal:  2006 
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Obj.  1 :  Provide  Necessary  Policy  &  Guidance  (cont.) 


4.  Establish  policy  to  require  documented  M&S  planning  at  the  joint  capability 
&  program  levels  as  part  of  the  Systems  Engineering  Plan,  T&E  Strategy 
and  T&E  Master  Plan 

Co-leads:  OUSD(AT&L)/DS,  ODOT&E;  Support:  Components 

Products:  Revised  policy  and  guidance  in  DoDI  5000.2,  DAG,  and  DOT&E  TEMP  Planning 
Guidance 

Completion  goal:  2006 

5.  Establish  guidelines  for  M&S-related  RFP  language  &  contract  provisions 

Lead:  OUSD(AT&L)/DS;  Support:  OUSD(AT&L)/DPAP,  Components 
Products:  Sample  language  in  DoD  publications  (e.g.,  DAG,  SEP  Preparation  Guide, 
Contracting  for  Systems  Engineering  Guidebook)  regarding  M&S  requirements,  data 
rights,  and  the  responsibilities  and  liabilities  of  parties  regarding  sharing  and  reuse 

Completion  goal:  2006 

6.  Publish  practical  guidelines  for  security  certification  of  M&S  activities  falling 
under  multiple  Information  Assurance  Defense  Accreditation  Authorities 

Lead:  OASD(NII);  Support:  OUSD(AT&L)/DS,  NSA 

Products:  Guidelines  published  in  DoD  8500. 2-H,  per  DoDI  8500.2  “Information  Assurance 
Implementation,”  Feb  6,  2003 

Completion  goal:  2007 
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Objective  2:  Enhance  the  Technical  Framework  for  M&S 

7.  Establish  a  framework  for  data  interchange-related  standards 

Lead:  OASD(NII);  Support:  OUSD(AT&L)/DS 
Products:  Revised  guidance  in  Nil  policy  documents 

Completion  goal:  2008 

8.  Develop  a  product  development  information  metamodel  &  associated 
metadata  extensions  to  the  DoD  Discovery  Metadata  Specification 

Lead:  OUSD(AT&L)/DS;  Support:  OASD(NII),  Components 
Products:  Revised  DDMS;  revised  guidance  in  DAG. 

Completion  goal:  2008 

9.  Support  development  of  open  commercial  systems  engineering-related 
standards,  such  as  OMG’s  Systems  Modeling  Language  (SysML)  and 
ISO  Standard  1 0303  AP-233 

Lead:  OUSD(AT&L)/DS;  Support:  DLA,  OUSD(AT&L)/DDRE,  OASD(NII) 

Products:  Published  standards  suitable  for  adoption  by  DoD 

Completion  goal:  2007 

10.  Establish  a  forum  to  clarify  the  characteristics  and  application  of  various 
distributed  simulation  standards  (HLA,  TENA,  DIS,  ALSP,  SI3,  etc.)  and 
examine  opportunities  for  convergence 

Lead:  OUSD(AT&L)/DDRE  Support:  OUSD(AT&L)/TRMC  &  DS,  ODOT&E,  Components 
Products:  (1)  Information  on  strengths  &  weaknesses  of  the  various  standards;  (2) 

agreement  on  policy  and/or  guidance  on  the  use  of  distributed  simulation  standards;  (3) 
a  way  ahead  regarding  distributed  simulation  standards 

Completion  goal:  2007 
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Obj.  2:  Enhance  the  Technical  Framework  for  M&S  (cont.) 


1 1 .  Improve  the  utility  of  the  DoD  Architecture  Framework  (DoDAF)  for 
acquisition 

11-1.  Develop  the  Acquisition  Overlay  (profile)  for  DoDAF  v2.0 
Lead:  OUSD(AT&L)/DS;  Support:  OASD(NII),  Components 
Products:  Acquisition  Overlay  for  DoDAF  v2.0 

Completion  goal:  2006 

1 1  -2.  Support  development  of  open  commercial  standards  for  the 
depiction  and  interchange  of  DoDAF-compliant  architectures 

Lead:  OASD(NII)  Support:  OUSD(AT&L)/DS 

Products:  Published  standards  suitable  for  adoption  by  DoD  in  DoDAF  2.0;  revised 
guidance  in  DAG 

Completion  goal:  2007 

12.  Establish  a  standard  template  of  key  characteristics  (metadata)  to 
describe  reusable  M&S  resources 

Lead:  OUSD(AT&L)/DDRE  Support:  OUSD(AT&L)/DS  &  TRMC,  DOT&E; 
Components 

Products:  Published  standard  template;  usage  guidance  in  DAG 

Completion  goal:  2007 
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Objective  3:  Improve  Model  &  Simulation  Capabilities 


13.  Establish  a  process  to  ensure  acquisition  needs  are  reflected  in  DoD 
M&S  priorities,  including  S&T 

Lead:  OUSD(AT&L)/DDRE;  Support:  OUSD(AT&L)/DS 

Products:  Incorporate  in  M&S  requirements  process  (ref:  DoD  M&S  Master  Plan)  a 
method  to  capture  and  prioritize  those  acquisition  needs. 

Completion  goal:  2007 

14.  Define  and  foster  best  practices  for  efficient  development  and  evolution 
of  credible  M&S  tools,  incorporating  user-defined  requirements,  a 
systems  engineering  approach,  and  appropriate  verification  &  validation 

Lead:  OUSD(AT&L)/DDRE;  Support:  OUSD(AT&L)/DS,  Components 
Products:  Best  practices  DMSO  publication,  available  via  MSIAC,  DTIC,  etc.;  DAG 
guidance  to  use 

Completion  goal:  2008 
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Obj  3:  Improve  Model  &  Simulation  Capabilities  (corn.) 


15.  Enable  readily-available  distributed  live-virtual-constructive  environments, 
leveraging  related  initiatives 

15-1 .  Establish  DoD-wide  standards  for  distributed  environments 

Lead:  OUSD(AT&L)/DDRE;  Support:  OUSD(AT&L)/TRMC  &  DS;  ODOT&E;  Components 
Products:  Published  standard;  DODI  (#  TBD)  policy  to  use 

Completion  goal:  2008 

15-2.  Make  candidate  simulations,  labs  and  ranges  compliant  with  these 
standards 

Lead:  Components;  Support:  OUSD(AT&L)/DS  &  TRMC 

Products:  Toolkit  of  live,  virtual  and  constructive  representations  ready  to  be  employed  in 
distributed  events 

Completion  goal:  2009 

15-3.  Provide  services  to  help  plan  and  conduct  distributed  events 

Lead:  Components;  Support:  OUSD(AT&L)/TRMC  &  DDRE,  DISA 
Products:  Fee-based  technical  services  to  help  users  (e.g.,  PMs,  Capability  Managers, 
OTAs)  plan  and  conduct  distributed  events 

Completion  goal:  2009 
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Obj  3:  Improve  Model  &  Simulation  Capabilities  (corn.) 


16.  Centrally  fund  and  manage  the  development  and  maintenance  of  high- 
priority,  broadly-needed  M&S  tools 

16-1 .  Identify  and  prioritize  broadly-needed  M&S  tools 

Lead:  OUSD(AT&L)/DDRE;  Support:  OUSD(AT&L)/DS;  Components 
Products:  Prioritized  list  of  common  M&S  tool  needs 

Completion  goal:  2007 

16-2.  Conduct  one  or  more  pilot  projects  to  develop  new  M&S  tools  or 
update  existing  ones  to  meet  these  needs 

Lead:  OUSD(AT&L)/DDRE;  Support:  OUSD(AT&L)/DS,  Components 
Products:  Proof  of  concept  for  managing  the  development/evolution  of  M&S  tools  to 
meet  broadly-shared  needs 

Completion  goal:  2009 

16-3.  Expand  the  scope  of  central  M&S  tool  management  as  warranted 
by  pilot  project  results  and  the  list  of  common  M&S  needs 

Lead:  OUSD(AT&L)/DDRE;  Support:  OUSD(AT&L)/DS,  Components 
Products:  Optimal  means  to  meet  common  needs  for  M&S  tools 

Completion  goal:  2011 
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Objective  4:  Improve  Model  &  Simulation  Use 

17.  Provide  potential  acquisition  M&S  users  the  knowledge  needed  to 
formulate  an  effective  M&S  strategy  via  ready  access  to  M&S  expertise  and 
information  about  M&S  capabilities  and  gaps,  reusable  resources,  lessons- 
learned,  etc. 

Lead:  OUSD(AT&L)/DS;  Support:  OUSD(AT&L)/DDRE 

Products:  Revised  guidance  in  DAG;  improved  knowledge  base  in  MSIAC;  assist  visits 
(e.g.,  by  OUSD(AT&L)/DS) 

Completion  goal:  2007 

18.  Define  best  practices  for  disciplined  M&S  planning  &  employment 

>  Rigorous  analysis  of  M&S  requirements  and  alternative  solutions,  selection  of  best 
course 

>  Efficient  configuration  management,  initialization,  execution  and  post-run  analysis 

>  Cautions  against  inappropriate  use;  approaches  to  maximize  cost-effective  reuse  across 
lifecycle 

Lead:  OUSD(AT&L)/DS,  Support:  OUSD(AT&L)/DDRE,  Components 
Product:  Revised  best  practices  guidance  in  DAG  and  MSIAC 

Completion  goal:  2007 
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Obj.  4:  Improve  Model  &  Simulation  Use  (cont.) 

19.  Facilitate  the  sharing  of  reusable  resources 

19-1.  Establish  a  DoD-wide  business  model  for  compensating  providers 
of  reusable  M&S  resources  (e.g.,  information,  software,  services) 

Lead:  OUSD(AT&L)/DDRE;  Support:  OUSD(AT&L)/DS,  OUSD(P&R),  OUSD(C)/PA&E, 
Components 

Product:  Documented  business  model;  revised  policy  and/or  guidance  in  DoD  5000  series 
and  DAG 

Completion  goal:  2007 

19-2.  Establish  DoD  policy  and/or  guidance  regarding  responsibilities  to 
share,  protect  and  properly  use  information  and  M&S  tools 

Co-Leads:  OASD(NII)  and  OUSD(AT&L)/DDRE;  Support:  OUSD(AT&L)/DS  &  DPAP, 
OUSD(P&R),  OUSD(C)/PA&E,  Components 

Product:  Revised  policy  and/or  guidance  in  various  issuances  (e.g.,  DoD  5000  series,  DAG, 
contracting  guidance) 

Completion  goal:  2007 

19-3.  Enhance  the  means  (e.g.,  directory  service,  registries,  bulletin 
boards)  to  discover  existence  of  reusable  M&S  resources  and  contact 
information 

Lead:  OUSD(AT&L)/DDRE  Support:  OUSD(AT&L)/DS,  OUSD(P&R),  OUSD(C)/PA&E, 
Components 

Product:  Functional  means,  with  appropriate  resources  and  incentives,  and  a  continuous 
improvement  process 

Completion  goal:  2007 
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Obj.  4:  Improve  Model  &  Simulation  Use  (cont.) 


20.  Define  the  types  of  information  DoD  organizations  shall  make  available  to 
others  with  a  valid  need  to  know  and  the  processes  to  obtain  them  (per 
reuse  business  model) 

20-1 .  Scenario  data 

Lead:  OUSD(AT&L)/DDRE  Support:  OCJCS(J8),  OUSD(C)/PA&E,  DIA,  Components 
Product:  Approved  scenarios  and  process  to  obtain 

Completion  goal:  2007 

20-2.  System-related  data 

Lead:  OUSD(AT&L)/DS;  Support:  Components 

Product:  Authoritative  system  data  (characteristics  and  performance,  interactions, 
interfaces,  logistic  support,  etc.)  and  process  to  obtain 

Completion  goal:  2007 

20-3.  Threat  data 

Lead:  DOD  MSEA  for  Threat  Data;  Support:  OUSD(AT&L)/DDRE  &  DS,  Components 
Product:  Authoritative  threat  data  and  process  to  obtain 

Completion  goal:  2007 

20-4.  Natural  environment  data 

Lead:  DoD  Natural  Environment  MSEAs;  Support:  OUSD(AT&L)/DDRE  &  DS, 
Components 

Product:  Authoritative  natural  environment  data  and  process  to  obtain 

Completion  goal:  2007 
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Obj.  4:  Improve  Model  &  Simulation  Use  (cont.) 


21 .  Foster  cost-effective  VV&A 

21-1.  Require  DoD-wide  standardized  documentation  of  VV&A 

Lead:  OUSD(AT&L)/DDRE;  Support:  OUSD(AT&L)/DS,  Components 
Products:  Revised  policy  in  DODI  5000.2  and  5000.61 ;  revised  guidance  in  DAG 

Completion  goal:  2007 

21-2.  Develop  risk-based  methodology  and  associated  guidelines  for 
VV&A  expenditures 

Lead:  OUSD(AT&L)/DDRE;  Support:  OUSD(AT&L)/DS,  Components 
Products:  Updated  DMSO  VV&A  Best  Practices  documents/web  site;  guidance  in 
DAG 

Completion  goal:  2006 

21-3.  Examine  a  program’s  VV&A  when  M&S  informs  major  acquisition 
decisions  and  unambiguously  state  the  purpose,  key  assumptions  and 
significant  limitations  of  each  model/simulation  when  results  are 
presented. 

Lead:  OUSD(AT&L)/DS  Support:  DoD  Components 

Products:  Guidance  &  training  for  oversight  personnel;  updates  to  DAG  Chaps  4  &  9 

Completion  goal:  2006 
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Obj.  4:  Improve  Model  &  Simulation  Use  (cont.) 


22.  Assess  the  use  of  COTS  systems  engineering  tools  (modeling 
environments)  for  collaborative  architecture  development 

Lead:  OUSD(AT&L)/DS;  Support:  OASD(NII),  Components 
Products:  Revised  guidance  in  DAG;  enhanced  M&S  body  of  knowledge  for 
dissemination 
Completion  goal:  2006 

23.  Define  and  capture  meaningful  metrics  for  M&S  utility  in  acquisition 

Lead:  Navy;  Support:  OUSD(AT&L)/DS,  Components 

Products:  Metric  definitions  in  DAG;  methods  to  capture  and  submit  data  in  DAG; 
data  from  individual  projects  in  MSIAC,  Body  of  Knowledge,  etc. 

Completion  goal:  2007 
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Objective  5:  Shape  the  Workforce 


24.  Define  required  M&S  competencies  for  the  acquisition  workforce 

Co-Leads:  DAU  and  OUSD(AT&L)/DS;  Support:  OUSD(P&R),  OUSD(AT&L)/DDRE, 
OUSD(C)/PA&E,  Components 

Product:  Identified  lead  FIPT;  workforce  qualification  requirements;  management  process 
&  structure 

Completion  goal:  2008 

25.  Harvest  lessons  from  commercial  sector  activities  in  the  use  of  M&S  to 
support  product  development 

Lead:  OUSD(AT&L)/DS;  Support:  OUSD(AT&L)/DDRE,  Components 
Products:  Lessons  collected  at  a  defined  site  (TBD);  annual  update  to  best  practices  in 
DAG  of  lessons  from  industry  that  should  be  considered  by  PMs  in  planning  for  M&S 
Completion  goal:  Recurring;  initial  in  2007 

26.  Assemble  and  evolve  the  M&S  Body  of  Knowledge  (information  set) 
relevant  to  acquisition 

Lead:  OUSD(AT&L)/DDRE;  Support:  OUSD(AT&L)/DS,  Components 

Product:  Information  base  available  to  potential  M&S  users  (e.g.,  PMs,  CMs,  OTAs); 

source  material  for  education  and  training 
Completion  goal:  Recurring;  initial  in  2007 
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Obj.  5:  Shape  the  Workforce  (corn.) 


27.  Drawing  on  the  M&S  Body  of  Knowledge,  educate  and  train  the  workforce 
to  achieve  required  M&S  competencies 

27-1.  Provide  M&S  knowledge  via  an  expanded  set  of  DAU  courses,  the 
Defense  Acquisition  Guide,  and  on-line  CLMs 

Lead:  DAU;  Support:  OUSD(AT&L)/DS  &  DDRE,  Components 

Product:  Expanded  set  of  DAU  courses,  improved  M&S  guidance  in  the  Defense 

Acquisition  Guide,  on  line  Continuous  Learning  Modules;  a  better  educated  workforce 

Completion  goal:  2009 

27-2.  Provide  M&S  knowledge  via  conferences,  workshops,  and  assist 
visits 

Lead:  OUSD(AT&L)/DS;  Support:  DAU,  OUSD(AT&L)/DDRE,  Components 
Product:  Annual  outreach  program;  a  better  educated  and  trained  workforce 
Completion  goal:  Recurring;  initial  in  2006 

28.  Improve  the  knowledge  and  expertise  available  through  the  MSIAC  to 
make  it  of  greater  utility  to  the  acquisition  community 

Lead:  OUSD(AT&L)/DDRE;  Support:  OUSD(AT&L)/DS,  OUSD(P&R),  OUSD(C)/PA&E, 
Components 

Product:  Plan  of  action  with  coordinated  MSIAC  CONOPS  &  staffing  requirement;  list  of 
knowledge  shortfalls  that  MSIAC  will  take  on;  success  criteria  &  process  to  bring  MSIAC  up 
to  criteria 

Completion  goal:  2008 
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Next  Steps 


□  Broadly  vet  actions  (DoD,  Industry) 

>  Fine-tune  actions,  lead  &  support,  products,  etc. 

□  Finalize  Acquisition  M&S  Master  Plan 

>  Acqn  M&S  Working  Group  consensus 

>  SE  Forum  approval  (Dec  2005) 

>  Informal  DoD  coordination 

>  Formal  coordination  as  a  DoD  issuance? 

>  USD(AT&L)  approve  plan 

□  Implement  plan,  monitor  action  completion 

□  Assess  impact  (metrics) 
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Discussion 

□  Questions  and  comments  are  invited  here  as  time  permits 

□  NDIA  M&S  Committee  meeting  1445-1730  Thursday  to 
answer  remaining  questions  and  discuss  your  change 
recommendations 

>  All  are  welcome  to  attend 
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BACKUP 
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Top-Down  Derivation/Cross-Check 


Desired  Acquisition 
Environment  per  CJCSI 
3170  and  DoDD  5000 


ft, 


Needed  Systems 
Engineering  Capabilities 


Characteristics  annotated  as  AE1,  AE2,  ...  AEn 


^  Annotated  as  SE1,  SE2, ...  SEn 


/> 


T\ 


M&S  Processes 
for  Better  SE 


\  ft, 


Annotated  as  MSI,  MS2, ...  MSn 


Gaps  in  Enabling  M&S 
Processes 


Annotated  as 
G1 ,  G2, ...  Gn 


\ 

Needed  Actions 

V 

/ 

Annotated  as 
A1,  A2,...An 
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Desired  Acquisition  Environment: 

Key  CJSCI  31 70.01  E  Policies 

AE1 

■  Jojnt  conceots-centric  capabilities  identification  process  to  allow  joint 
forces  to  meet  the  full  range  of  military  operations  and  challenges. . . 

■  Assess  existing  ana  proposed  capabilities  in  light  of  their  contribution 
to  future  joint  allied  and  coalition  operations. . . .  Produce  capability 
proposals  that  consider  the  full  range  of,  DOTMLPF_  solutions  in  order 
to  advance  joint  warfighting  in  a  unilateral  and  multinational  context. 

■  New  solution  sets. . .  crafted  to  deliver  technologically  sound,  testable. 
AE4  sustainable  ancj  affordable  increments  of  militarily  useful  capability. 

AE5 

■  The  FoS  and_  SoS  solutions  may  also  require  systems  delivered  by 
rmjjtipje  sponsors/rnaterjej  developers. AE6 

■  The  process  to  identify  capability  gaps  and  potential  solutions  must  be 
supported  by  a  robust  analytical  process  AE7 

■  JCIDS  implements  a  capabilities-based approach  that. ..requires  a 
^collaborative  process  that  utilizesjoint  concepts  and  integrated 
ae9  architectures  to  identify  prioritized  capability  gaps  and  integrated 

DOTMLPF  and  policy  approaches  to  resolve  those  gaps 

AE11 
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Desired  Acquisition  Environment: 

DoDD  5000.1  Acquisition  Policies 

AE12 

“The  primary  objective  of  Defense  acquisition  is  to  acquire  quality  products  that 
satisfy  user  needs  with  measurable  improvements  to  mission  capability  and 
operational  support,  in  a  timely  manner,  and  at  a  fair  and  reasonable  price.  ” 

AE13  AE14  AE15 

Governing  policies: 

>  Flexibility,  Responsiveness  (time-phased  capabilities,  evolutionary 
acquisition).  Innovation,  Discipline.  Streamlined  Effective  Management 

AE18  AE19 

>  Armaments  Cooperation;  Collaboration:  Competition;  Cost  and 
Affordability:  Cost  Realism:  Cost  Sharing;  Financial  Management; 
Independent  OTAs;  Information  Assurance;  Integration  Supepjggty; 

AE20  Integrated  T&E;  Intelligence  Support;  Interoperability;  Kno wleaoe-Based 
Acquisition:  Legal  Compliance;  Performance-Based  Acquisition; 

AE23  Performance-Based  Logistics:  Products  Services  and  Technologies  [seek 
most  cost-effective  solution  over  the  system's  Ijfe  cycle],  Professional 
Workforce,  Program  Information  [complete,  current,  tailored];  Program 
Stability;  R&D  Protection;  Safety;  Small  Business  Participation;  Software 
Intensive  Systems;  Streamlined  Organizations;  Systems  Engineering: 
AE26Technoloov  Development  and  Transition:  Total  Systems  Approach  ae27 


>  Oct  04  policy  memo:  Technical  reviews . . .  shall  be  event-driven  ae28 
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Necessary  Systems  Engineering  Capabilities 

(which  M&S  can  affect;  derived  from  Desired  Acquisition  Environment) 

SE1.  Early,  continuing  systems  engineering  from  an  SoS/FoS  capabilities 
perspective;  seamless  transition  from  JCIDS  to  acquisition 

(AE1  -3, 5, 9-1 1 ,1 6,20,21 ,25,27) 

SE2.  Lifecycle-wide  exploration  of  the  maximum  available  trade  space, 
including  time-phased  requirements  and  technology  insertion 

(AE1  -5,7,1 0,1 1 ,1 3,1 6,1 9,23-27) 

SE3.  Collaboration  among  all  stake  holders  (multiple  gov’t  and  contractor 
organizations)  for  key  enterprise-level  SE  decisions  (AE6-8, 10,18,22, 25, 27) 

SE4.  Rapid  assessment  of  concept/design  alternatives  <ae2, 4, 7, 10, 14,16,19,25,28) 

SE5.  Comprehensive,  accurate,  event-based  assessment  of  technical 
baselines;  avoidance  of  costly  fixes  for  problems  discovered  late 

( AE2-4,7,9,1 0,1 2-1 7,1 9,20,22,24-26,28) 

SE6.  Focused,  effective  &  efficient  testing;  including  at  the  capability  level 

(AE1 ,2, 4, 5, 9-1 1 ,13,15,19-22,25) 

SE7.  Appropriate  reuse  of  all  resources  -  information,  software  tools, 
expertise,  facilities,  ranges,  etc.  -  across  programs  &  organizations 

(AE4,1 4,1 5,1 9,24,25) 
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(Jl)M&S  Processes  for  Better  Systems  Engineering 

(derived  from  Needed  Systems  Engineering  Capabilities) 

MSI.  Use  of  a  model-based  systems  engineering  approach  (sei,2,4,5) 

(Emerging  concept  under  INCOSE,  OMG,  etc.;  growing  suite  of  COTS  SE  tools) 

>  SE  tool  modeling  environments  to  analyze  requirements,  develop 
architecture,  and  specify  constraints;  views  linked  to,  and  generated 
from,  an  underlying  database 

>  Embedded  simulation  to  verify  the  architecture  and  assess  its  merits 

>  Automated  generation  of  documents/reports 


Capability 
System 

Engineering  | 


Coordinate/support  Development  and  Engineering  Changes 


MS2.  Establishing  M&S-enabled  collaborative  engineering  environments 

(SE1,2,3,4,5,6) 

>  Shared,  authoritative  information 

>  Interoperable  modeling  environ¬ 
ments  for  architecting/design 

■  Models  as  communication  means 

>  Models  &  simulations  to  assess 

■  Option  to  immerse  warfighters,  etc. 

>  Distributed  live-virtual-constructive 
environments  for  integration,  verification,  and  test 


Program 

System 

Engineering 


x 


System 

T&E 


MS3.  Model-Test-Fix-Model  process  across  the  life-cycle  (SE4,s,6) 

>  Better  test  planning,  more  effective  tests 

>  Increased  M&S  validity;  credible  surrogates;  reuse  savings 
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M&S  Processes  for  Better  Systems  Engineering 

MS4.  Harnessing  M&S  knowledge  to  formulate  an  effective  M&S  strategy 

(SE2,3,4,5,7) 

>  Ready  access  to  M&S  expertise  and  information  about  capabilities 
and  gaps  (M&S  holes),  reusable  resources,  lessons-learned,  etc. 

MS5.  Disciplined  M&S  planning  &  employment  <se2,4,5,7) 

>  Rigorous  analysis  of  M&S  requirements,  alternatives,  best  course 

>  Efficient  configuration/initialization,  execution  and  post-run  analysis 

>  Avoid  inappropriate  use;  maximize  cost-effective  reuse  across  lifecycle 

MS6.  Efficient  development/evolution  of  credible  M&S  tools  (SE2,3,5,7) 

>  A  systems  engineering  approach  with  appropriate  V&V 

MS7.  Access/sharing  of  authoritative,  understandable  data  needed  for  M&S 
representations  <se2,3,4,5,7) 

>  Reducing  a  major  time  and  cost  burden  that  inhibits  M&S  use 

MS8.  Inspection  of  M&S  used  to  inform  acquisition  decisions  <se2,s,7) 

>  Examine  capabilities  and  limitations  (VV&A)  of  M&S 

>  During  lead-up  to  program/technical  reviews,  OTRRs,  DABs,  etc. 


37 


D2 


Gaps 


1.  Management 


Gl.  Robust  but  confused  landscape  of  M&S  activities;  no  clearly 
designated  leadership  or  effective  coordinating  mechanism  (msi-8) 

>  Current  EXCIMS  ineffective;  little  coordination  for  capabilities/SoS/FoS 


G2.  Inadequate  constancy  of  purpose  because  time  to  fix  problems  » tour 
length;  “DoD  has  an  attention  deficit  disorder”  <ms2-7) 

G3.  Gov’t  acquisition  guidelines  don’t  promote  M&S  use  or  reuse  (msi-6) 

G4.  No  DoD  requirement  for  formal  M&S  planning  to  support  acquisition 
(other  than  T&E)  <msi-5) 

G5.  No  contractual  guidelines  regarding  M&S  and  the  data  it  needs  <msi-8) 


G6.  Gov’t  typically  doesn’t  give  contractors  meaningful  M&S  guidance 

(MSI  ,2,6,8) 


G7.  Most  DoD  M&S  takes  a  project,  vice  an  enterprise,  approach  <ms2,3,6,7) 
G8.  No  consensus  on  value  of  integrated  architectures,  nor  responsibility 

for  (MSI ,2) 


G9.  Managing  distributed  collaboration  is  very  hard  (msi-8) 


G10.  Public  law  precludes  OT  based  solely  on  M&S,  but  no  clear  guidance 
on  use  for  SoS/FoS  T&E  <ms2,3,5,6,8) 
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D2 


Gaps 


2.  Architecture/standards/technical  framework 

Gil.  No  standard  modeling  notation  (like  UML  v2.0)  for  capturing  full  range 
of  information  critical  to  system  engineering  (e.g.,  structure,  behavior, 
requirements  hierarchy/traceability,  test  cases,  verification  results)  (msi ,2,6,7) 

G12.  No  standard  for  interchanging  systems  engineering  information  (same 
examples  as  above)  (msi  ,2,6,7) 

G13.  No  conceptual  framework  (like  Open  System  Interconnect  protocol  stack) 
for  data  interchange  (msi,2,3,6,7) 

G14.  Lack  of  agreement  on  a  common  distributed  simulation  standard 
increases  complexity  and  cost,  limits  simulation  interoperability  <ms2,5,6) 

G15.  DoDAF  vl  .0  is  difficult  to  use  for  architecting  due  to  lack  of  data- 
centricity  and  executability;  some  products  of  marginal  value<Msi,2,6,7) 

G16.  Use  of  DoD-unique  standards  limits  their  user  base,  quality,  COTS  tool 
support,  and  opportunities  for  reuse  (msi ,2,5,6) 
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D2 


Gaps 

3.  Model/simulation  capabilities  &  use 

G17.  Many  M&S  tool  gaps  and  deficiencies  <msi,2,3,5,7) 

>  What’s  modeled  (e.g.,  urban  warfare,  comm  networks,  threats,  system  sustainment) 

>  Fidelity,  granularity,  interoperability 

>  Only  limited  consensus  on  common  models  to  be  used  across  a  domain 

G18.  No  good  way  to  develop  and  maintain  widely-needed  M&S  tools  that  cut 
across  programs  (mss,6) 

>  Not  incorporating  mods  by  other  organizations  into  “street  version,”  etc. 

G19.  M&S  developers,  not  M&S  users,  tend  to  drive  M&S  development  <ms6) 

G20.  In  general,  architecture  development  (modeling)  is  lagging,  not 

collaborative,  and  not  exploiting  COTS  SE  tools  (modeling  environments) 

(MSI  ,2) 

G21.  No  readily-available  distributed  M&S  infrastructure  (e.g.,  JDEP)  <ms2,5) 

G22.  Hard  to  get  security  certification  for  multi-organization  (company/ 
Service)  distributed  simulation  <ms2,3,5,6) 

G23.  Hard  to  get  approval  and  security  certification  for  M&S  involving 
multiple  compartmented  programs  (SAPs)  <ms2,3,5,6,7) 
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D2 


Gaps 


4.  Trustworthiness/VV&A 

G24.  Post-development  model  validation  expensive  and  slow  <ms2,3,5,8) 

G25.  VV&A  often  weak  or  non-existent;  documentation  inconsistent 

(MS2,3,5,8) 

>  Plans  to  use  M&S  to  avoid  testing  costs  often  rejected  due  to  poor/no 
validation 

G26.  VV&A  usually  not  enforced  and  also  not  examined  during  program 
reviews  (MS2,3,5,6,8) 

G27.  Models  and  sims  often  not  updated  to  reflect  empirical  evidence 
(e.g.,  test  results)  <ms2,3,5,8) 
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D2 


Gaps 

5.  Sharinq/reuse  and  protection  of  tools  &  information 

G28.  Little  reuse;  only  7%  of  models  &  sims  used  on  >1  program  (MS2,5,6) 

G29.  Concurrent  engineering  requires  an  integrated  process,  data  sharing 
and  a  coherent  tool  set,  but  <20%  of  programs  have  such  a  collaborative 
environment  <ms2,7) 

G30.  Hard  to  discover  reusable  resources  (software,  info,  services)  (MS2,4,5,7) 

>  M&S  repositories  are  not  integrated,  lack  an  effective  search 
capability,  and  are  mostly  empty 

>  MS  I  AC  knowledge/expertise  is  lacking 

G31.  Insufficient  info  (metadata)  to  evaluate  data/reuse  candidates  (ms2,4,5,7) 

G32.  Hard  to  obtain  reusable  resources  <ms2,4,5,7) 

>  Industry  to  gov’t:  To  protect  proprietary  info  &  competitive  advantage 

>  Gov’t  to  industry:  Contractual  liabilities  associated  with  GFE/GFI 

>  Gov’t  to  gov’t:  Concerns  about  misuse;  cost  to  deliver  and  guide 

G33.  No  incentives  to  encourage  reuse  <ms2,3,5,6) 

>  Negative  incentives  include  cost  to  make  reusable,  workload 
assisting  users,  vulnerability  to  criticism 

[plus  approval  and  security  certification  gaps  22  &  23  listed  under  M&S  use] 
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D2 


Gaps 


6.  Research/S&T/tech  base 

G34.  Conceptual  foundation  of  M&S  weak  (mss,6) 

>  E.g.,  theoretical  understanding  of  modern  warfare,  human 
behavior,  relating  M&S  at  different  granularities,  dealing  with 
uncertainty,  agent-based  modeling  and  generative  analysis 

G35.  Little  acquisition  community  input  to  DoD  S&T  management 
regarding  needed  M&S-related  research  (ms2,s,6) 

7.  Business  model,  metrics  &  ROI.  funding  and  incentives 

G36.  No  business  model  for  how  M&S  capabilities  should  be  developed, 
used  and  maintained  <msi-8) 

G37.  Metrics  are  critical  to  keep  interest  and  funding  up,  but  metrics 
regarding  M&S  use  and  cost-effectiveness  are  inadequate  <msi-8) 

>  M&S  funding  difficult  to  identify;  most  embedded  within  other  PEs 

G38.  Too  little  funding  (MS2-7) 
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D2 


Gaps 


8.  Workforce  Shaping 

G39.  Body  of  knowledge  for  M&S  support  to  acquisition  is  deficient,  not 
managed  <msi,2,4-6,8) 

G40.  Acqn  community  managers  and  staffs  mostly  uninformed  about 
M&S  capabilities  and  limitations  <msi-8> 

>  Weak  acquisition  personnel  understanding  of  commercial  M&S 
activities  (“We  don’t  get  out  enough”) 

>  Not  enough  M&S  experts  (no  career  path  [except  Army],  no 
formal  education  or  training) 

G41.  M&S  developers  lack  understanding  of  modeling  best  practices, 
abstraction  techniques,  context  dependencies,  etc.  <ms3,6) 

G42.  M&S  users  often  not  adequately  trained  <msi ,2, 4, 5, 8) 

G43.  Insufficient  M&S  education  options  <ms2,4,5,6,8) 
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**■  Aging  Transport  Systems 

Rulemaking  Advisory  Committee 

>> 1  nitially  chartered  19  J  anuary  1999 

>■  Re-chartered: 

25  J  anuary  2001 
28  J  anuary  2003 
21 J  anuary  2005 

>-  See  www.  mitrecaasd .  org/  atsrac/  i  ndex.  html 


26  October  2005 


NDIA  Systems  Engineering  Conference 


2 


ATSRAC 


national  USA  attention  to  aviation  safety 

Growing  number  of  flights  will  create 
more  accidents,  even  with  a  lower  rate 

White  House  Commission  on  Aviation 
Safety  and  Security  -  22  August  1996 


26  October  2005 


NDIA  Systems  Engineering  Conference 
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ATSRAC 


port 


Final  Report  published  12  February  1997 


1.9  I  n  cooperation  with  airlines  and 
manufacturers,  the  FAA's  Aging  Aircraft 
Program  should  be  expanded  to  cover 
non- structural  systems 


26  October  2005 


NDIA  Systems  Engineering  Conference 
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ATSRAC 


nSnaEff  atsrac  Emets 


Chairman:  Kent  Hollinger 


FAA 
0-  DoD 
0-  ALPA 
•E  ATA 
•E  AIA 
0-  NASA 
0-  GAMA 

0-  Flight  Safety  Foundation 


•E  JAA 

•*-  T ransport  Canada 
+  AECMA 

•¥  General  J  ohn  Loh 
0-  Boeing 
0-  Airbus 
•E  NEMA 
4-  SAE  (DuPont) 


26  October  2005 


NDIA  Systems  Engineering  Conference 


International  Federation  of  Airworthiness 
0-  NADA/F 
NBAA 

Northwest  Airlines 
Garrett  Aviation  Services 


26  October  2005 


NDIA  Systems  Engineering  Conference 
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ATSRAC 


JnlibaJ  Five 


Sampling  inspection  of  the  fleet 
Review  of  fleet  service  history 
I  mprovement  of  maintenance  criteria 
Review  and  update  standard  practices  for  wiring 

*¥  Review  air  carrier  and  repair  station  inspection 
and  repair  training  programs  and  recommend 
actions  to  address  aging  systems 


26  October  2005 


NDIA  Systems  Engineering  Conference 
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ATSRAC 


tat  us  or  J  njtj 


All  tasks  are  complete 


■ 

ATSRAC  approval  of  Final  Reports 
during  J  anuary  2001  meeting 


Final  Reports  with  recommendations 
forwarded  to  FAA 


26  October  2005 


NDIA  Systems  Engineering  Conference 
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ATSRAC 


PTffl 


rj 


□cl  _l  r\ 


Non- intrusive  I  nspections  of  81  aircraft 


i.gg5flWg!  gij 


'icjffl 


3  -3./P.  H  n  dTVi  cIqsiLeIjbcj 


92,000  cycles 


ancies  (sorted  by  risk) 


irini 


its  f\ 


et-wlde  Safety  of  Flight  Concern  -  0 


PotentiMTfla^SFd*orTsrequently  Occurring  Item  -  182 
T-SB’s  reqifred  for  3  and  enhanced  inspection  guidelines  for  2 


Defects  noted  -  3,190 


26  October  2005 


NDIA  Systems  Engineering  Conference 


9 


ATSRAC 


ms  i 


[cdm. 


nued) 


I  ntrusive  I  nspections  of  6  retired  aircraft 


Acied  up  to  'ijOPAA'1  ,aiSh' 
Targeted  sos^rrlc^vji 


ipi  iuui  ^and  100,017  cydes 
ire  types  and  ai  rcraft  zones 


Re< 


TEuaunspection  and  NDT 
I/ed  cWa  2$  laboratory  tests  applied 


1  generw-  education  of  inspection  &  maintenance  personnel 
9  research  ^degradation!  chaffing^contami  nation,  NDT,  AFCB 
90  specific  -  splices, Pheat  shields* clamping 
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QR.CEE? 


Typical  Aircraft  Zo>n 
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ATSRAC 


ATSRAC 


Further  Jnforrnspdr 


I  ntrusive  I  inspection  Report  available  at: 


http:  //www.  mitrecaasd .  org/  atsrac/  i  ntrusive 
_inspection.html 
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Reviewed  79  ADs  with  repetitive  inspections 

>■■■•  Recommended  8  for  terminating  action 
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N  ext  F  o  uni aska  ( 


ATSRAC 


LI) 


Federal  Register  29  May  2001  p.  29203 


s 


ation^q  u  i  rements 


inai  [content  of  SWPM 


Ima 


ing  program  for  wire  systems 


>> Enhanced  maintenance  criteria  for  systems 
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ATSRAC 


eatus  of  Ph a 


All  tasks  are  complete 


Final  Reports  with  recommendations 
forwarded  to  FAA  in  2002 


♦^-Small  airplane  recommendations 
forwarded  to  FAA  in  J  anuary  2003 
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TH 


5 fSf 


]- 


ATSRAC 


Created  new  FAR  25  Subpart  H  for 
Electrical  Wiring  I  nterconnection  Systems 

i  ^Bedg^ol^ntlwinng  j"eg  u  I  ati  ons 
seplfetiora*  ** 

rej^iflcStioh  , 

s^ltern.eafety  assessments 


»>  Revised  current  FAR  25  sections  and 
created  Advisory  Circulars 
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ATSRAC 


Standard  Wire  Practices  Manual  (ESWPM) 

Defined  standard  format  for  new  ESWPM 

Created  a  Master  Breakdown  I  ndex  (MBI ) 
for  use  with  existing  ESWPM 
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ATSRAC 


Task  S  Results 


Created  an  Advisory  Circular  titled 
“Aircraft  Electrical  Wiring  interconnection 
Systems  Training  Program  " 

»*■ Applicable  to  air  carriers,  maintenance 
providers,  OEMs  and  STC  holders 

Voluntary  incorporation  is  encouraged 


26  October  2005 


NDIA  Systems  Engineering  Conference 


25 


ATSRAC 


task  9  Results 


SFAR  to  require  OEMs  to  implement  and 
communicate  an  Enhanced  Zonal 
Analysis  Program  (EZAP)  to  airlines 

Created  Special  Maintenance  Program 
Requirements  for  >30  seat  airplanes 

4- Created  new  FARs  to  require  training  on 
electrical  systems  (see  AC  from  Task  8) 
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NDIA  Systems  Engineering  Conference 


26 


ATSRAC 


Current  HVVG# 


Numerous  recommendations  to  FAA: 


bfidusct/  s]; 


kji 


BHHBM  for  EWI S  routing 
1 1 1  i  l  kji  ^ <  akers  (cyding  by  industry) 

hMf^^x^BreScJrs  (AFCB) 
^tfrectSjoFadditives  on  wiring  qualification 
>> Mamtenah^tra i niftg  req u i rements 
^Improved  shield  terminators 


>-Maintenad 
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ATSRAC 


Current  Schedule 


Notice  of  Proposed  Rule  Making  (NPRM) 
scheduled  for  6  October  2005 

Final  Rules  and  Advisory  Circulars  are 
expected  by  year  end  2006 

nclusion  of  airplanes  with  <30  seats  is 
required  for  new  designs  but  is  still  under 
consideration  retroactively  (ATSRAC  split) 


26  October  2005 


NDIA  Systems  Engineering  Conference 
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ATSRAC 


10  -  12  J  anuary  2006  (a)  ATA 


4-4-6  April  2006  @  Airbus  North  America 
4-  Sunset  of  ATSRAC 


26  October  2005 
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ATSRAC 


Questions? 
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Out  of  the  Ordinary:  Finding  Hidden  Threats 
by  Analyzing  Unusual  Behavior 

A  New  Approach  to  “Connecting  the  Dots”  in  Intelligence 

John  Hollywood 
RAND 

Presentation  to  the  NDIA  Systems  Engineering  Conference 

October  26,  2005 


Introduction 


*  Today’s  presentation  is  on  the  “Atypical  Signal  Analysis  and  Processing” 
(ASAP)  Schema,  an  architecture  for — 

-  Identifying  “unusual”  observations  worth  further  attention 

•  E.g,  violations  of  the  status  quo 

-  Relating  these  observations  to  other  data 

-  Generating  and  testing  hypotheses  about  these  observations 

Purpose  of  research  was  to  identify  and  characterize  terrorist  threats, 
but  basic  approach  may  have  a  number  of  other  applications 

-  Previously  asked  about  applications  for  “tracking  and  identifying 
problem  acquisition  personnel  early  in  the  process” 

-  Might  also  have  applications  for  PM  and  SE 

*  From  a  RAND  IR&D  project  -  results  documented  Out  of  the  Ordinary: 
Finding  Hidden  Threats  by  Analyzing  Unusual  Behavior  (RAND  MG- 
126-RC) 

-  Focuses  primarily  on  top-level  architecture,  with  some  discussion  of 
data  management  and  analysis  algorithms 

RAND  Unclassified  2 


Motivation  for  “Out  of  the  Ordinary” 

Research 


#  Immediate  motivation  was  to  improve  ability  to  “connect  the  dots”  -  pieces  of 
information  that  could  be  combined  to  produce  understanding  of  a  threat 

•  Focus  on  intelligence  and  homeland  security  data,  especially  on  reports 

-  Others  were  working  on  large-scale  data  mining;  taking  advantage  of 
intelligence  /  HLS  reports  was  unique  opportunity 

-  Also  wanted  to  avoid  privacy  /  legal  issues 

-  For  acquisition,  equivalent  would  be  steady  stream  of  reports  and 
products  coming  out  of  PM  and  acquisition  offices,  plus  any  reported 
complaints 

♦  Additional  focus  -  look  for  “anti-patterns” 

-  Knew  we  wanted  to  avoid  “fighting  the  last  war  problem”  (l.e.  just  looking 
for  patterns  matching  previous  attacks) 

-  Also  inspired  by  how  proactive,  successful  problem  solvers  have 
“connected  the  dots”  in  the  past... 


RAND 


Unclassified 
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Proactive,  Successful  Problem  Solvers 
Study  the  Atypical  to  “Connect  the  Dots” 


1.  Identify 
Atypical  Behavior 


2.  Learn  More  About 
the  Behavior 


3.  Understand  the 
Meaning  of  the  Behavior 


Plant  Managers’ 
E-mail  Traffic 


1 .  Observe  behavior, 
looking  for  status  quo 
deviations 


4.  Employ  risk- 
avoidance  and 
mitigation  heuristics 


4.  Respond  to 
the  Meaning 


Source:  Ethnographic  research  by  Prof.  K.  N.  McKay 


RAND 
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The  ASAP  Schema  Is  Modeled  After  Problem 
Solvers’  Analysis  of  the  “Out  of  the  Ordinary” 


Incoming  info 

•  Intelligence 
community 
data 

•  “Of  interest” 
security 
observations 


Link 
observations 
and  related 


Generate 
hypotheses 
about  the 


Test  the 
hypotheses 


Flag  the 
most 
significant 


Analysts 
review  the 
results  & 
provide 
feedback 


ASAP  analysis  is  an  iterative,  multi-directional  process 


Follow-up 

responses 


Improve  awareness  and  collaboration  for  networks  of 
intelligence  collectors  and  analysts  at  all  levels —  national, 

strategic,  operational  and  tactical 


RAND 
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Observations  on  Research  and 
Development  Needs 


*  Initial  research  focus  was  going  to  be  on  algorithms  development, 
but  numerous  tools  carrying  out  specific  tasks  to  “connect  the  dots” 
already  exist 

-  Text  to  structured  data  tools 

-  Network  analysis  tools  /  link  analysis  tools 

-  Data  mining  tools 

-  Collaboration  /  groupware  suites 

•  What  was  missing  was  putting  them  together  into  a  single 
architecture 

-  Led  to  focus  on  developing  ASAP  architecture,  rather  than 
specific  algorithms  or  tools 

•  What  is  also  needed  is  a  basic  suite  of  analysis  and  collaboration 
tools  that  can  be  deployed  quickly  and  cheaply,  and  can  be 
customized  and  tailored  to  the  situation  at  hand 

*  “Artificial  intelligence”  research  on  developing  models  of  the  status 
quo,  and  developing  algorithms  to  find  outliers,  also  needed 


RAND 
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The  ASAP  Architecture  for  Automated 
Support  to  Counter-Terrorism  Analysis 


Smart 

filtering 


J 


Data  of  interest 
on  “watched 
entities” 


Data  /  results 
to  review 


Intelligence  data 


Homeland  security  data 


Updated  filtering  rules 

Process  control 

•Review  new  data  /  results  and  determine  next  steps 
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ASAP  information  pool 
•Linked,  filtered  data 
•Analysis  results 
•Analysts’  observations  and 
communications 


75 

c 

o 

(0 

(/> 

(0 

"D 

0) 

Q_ 

Find 

“dots” 

(atypical 

data) 

Link 

data 

elements 

Generate 
and  test 
hypotheses 

f 

Analysis  results  (automated  and  manual) 


Requests  for  personal  data  (subpoena  process) 


•Analysis  is  iterative;  significant  results  “bubble  up” 

Data  gathering  and  analysis  procedures  adapt  to  new  findings 
•“Data  mine  as  needed;”  private  data  used  only  if  subpoenaed 


significant  Analysts’ 


results 


follow-up 

requests 


Analysts  review 
and  collaborate 
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Features  of  the  ASAP  Architecture 


Identifies  phenomena  that  are  “out  of  the  ordinary,”  even  if  they 
do  not  match  previous  patterns  for  suspicious  behavior 

Focuses  on  information  already  collected  from  intelligence  and 
other  government  monitoring  systems 

-  In  the  acquisition  /  SE  context,  this  would  be  acquisition  and 
SE-related  products,  and  any  incoming  complaints 

Employs  rules  allowing  data  to  be  analyzed  in  the  context  of 
existing  knowledge 

Employs  rules  that  are  dynamic  -  significant  developments 
automatically  increase  the  priority  of  related  information 

Explicitly  deals  with  uncertainty  by  formulating  and  testing 
competing  hypotheses 

Initiates  collaboration  of  personnel  needed  to  “connect  the  dots” 

-  Analysts’  reviews  are  considered  data  just  as  much  as 
source  information 


Unclassified 


An  Example  Scenario:  “Something  Bad 
Happened  on  November  9th” 


2/4:  24  high-speed  tuna  boats 
ordered  by  Panamanian  firm, 
destination  Singapore 


Seattle 


10/10:  Panamanian  company  leases  warehouse  in 
Philadelphia  wharf 

10/11:  Routine  inspection  identifies  tuna  boats  off-loading 
heavy,  odd-smelling  crates  (claim:  Panamanian  company 
moving  teakwood  for  furniture  manufacturing) 

11/5:  Tuna  boat  captains  found  to  have  no  prior  fishing, 
furniture,  or  shipping  related  connections;  two  captains 
previously  worked  for  Seattle  tuna-boat  building  firm. 


9/8:  Panamanian  company  applies  for 
18  tuna  boat  berths 
10/4:  Panamanian  company  reported 
pressuring  local  officials  for  special 
berthing  privileges  w/o  required 


8/4:  24  tuna  boats  arrive 
in  Sydney 

10/2:  Hiring  and  work 
oddities  reported  w/ 
respect  to  tuna  boats 
owned  by  a  Panamanian 
company 


RAND 
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Applying  the  ASAP  Schema  to  the  1 1/9  Scenario 


2/4:  24  tuna  boats  ordered 
by  Panamanian  firm, 
destination  Singapore 


Seattle 


Import  /  Export  Agent 

•Destination  mismatch :  Why  did  24  tuna  boats  arrive  in  Sydney  rather  than  Singapore? 
•Count  mismatch :  When  the  boats  did  get  to  Singapore,  why  are  there  only  18? 
•Unusual pressure:  Why  is  the  tuna-boat  shipping  company  pressuring  local  officials? 


Philadelphia 


Linking  Agent 
•Same  Panamanian  company 
is  generating  numerous  “out 
of  the  ordinary”  hits  in 
multiple  countries 


10/10:  Panamanian  company  leases  warehouse  in 
Philadelphia  wharf 

10/11:  Routine  inspection  identifies  tuna  boats  off-loading 
heavy,  odd-smelling  crates  (claim:  Panamanian  company 
moving  teakwood  for  furniture  manufacturing) 

11/5:  Tuna  boat  captains  found  to  have  no  prior  fishing, 
furniture,  or  shipping  related  connections;  two  captains 
previously  worked  for  Seattle  tuna-boat  building  firm. 


Commerce  Agent 

•Unexplained  traffic :  why  are  there  sudden  increases  in  tuna  boat  traffic  in 
Sydney,  Singapore,  and  Philadelphia  without  matching  increases  in  demand? 
•New  shipping  type  &  new  packaging  type :  teakwood  shipped  in  odd  containers 
via  tuna  boats  has  not  been  seen. 

•Hiring  mismatch :  why  is  the  Panamanian  company  hiring  captains  without 
experience? 


9/8:  Panamanian  company  applies  for 
18  tuna  boat  berths 
10/4:  Panamanian  company  reported 
pressuring  local  officials  for  special 
berthing  privileges  w/o  required 
paperwork 


8/4:  24  tuna  boats  arrive 
in  Sydney 

10/2:  Hiring  and  work 
oddities  reported  w/ 
respect  to  tuna  boats 
owned  by  a  Panamanian 
company 
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What  Can  Be  Done  in  the  Near  Term  To  Bring 
About  ASAP  Schema  Benefits?  (1) 


Basic  Collaboration  Support: 

&  Core  functionality  -  improve  collaboration 

-  Create  and  distribute  2-4  page  profiles  of  ordinary  behavior,  and 
examples  of  “significant”  violations 

•  Tight  initial  focus  on  training 

•  Update  regularly  -  “be  on  the  look  out  for...” 

-  Create  a  family  of  posting  boards  for  the  Community 

•  Have  unmoderated  boards  for  posting  “curious”  or  “unusual” 
information 


•  Have  moderated  board  for  posting  of  “key”  information  on  various 
potential  problems 

•  Have  weblogs  /  wikis  for  presenting  narratives 

*  Extended  functionality  -  support  analysis  of  posting  boards 

-  Allow  Google-like  search  engines  for  posts 

-  Use  simple  heuristics  on  posts,  looking  for  connections  and  patterns 
across  the  posted  messages 

Unclassified  1 1 
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What  Can  Be  Done  in  the  Near  Term  To  Bring 
About  ASAP  Schema  Benefits?  (2) 

Analysis  Support: 

*  Focus  on  evolutionary  build-out  of  analysis  tools,  starting  with  existing 
products 

&  Core  functionality  -  create  and  manage  “the  dots”: 

-  Network  database  /  analysis  tool  that  stores  data  in  a  replication- 
friendly  and  update-friendly  format 

-  Peer-to-peer  system  for  performing  the  distribution  and  updating 

•  Ideally,  includes  project  management  /  workflow  tools  as  well 

*  Extended  functionality  -  automatically  populate  and  analyze  “the  dots”: 

-  Free  text  /  data  to  structured  data  tool 

-  Advanced  link  analysis  tool 

-  Hypothesis  generation  tools  (pattern-matching  and  pattern-violation 
agents) 

-  Hypothesis  tracking  and  testing  tools 


RAND 


Unclassified 
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What  Might  a  Near-Term  ASAP  Network 

Look  Like? 


Info 
providers 


Info 
customers 


Analysts 


GIS 


■ 

■ 

hypotheses 


Text-to- 
structured  data 


Identify 
Advanced  hypotheses 
link 

analysis 


RAND 
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How  Might  The  ASAP  Schema  Be 
Applied  to  Defense  Acquisition? 

Generate  simple  models  of  the  “status  quo”  for  acquisition  programs 
and  acquisition  jobs 

-  Describe  major  steps  that  the  program  or  person  performing  the 
job  should  go  through,  and  what  each  step  should  look  like 

•  Acquisition  Culture,  Successful  Program  Implementations 
discussions  may  be  useful  here 

•  Exercise:  “this  quarter,  the  program  does  nothing  problematic. 
What  kind  of  things  occur?” 

ID  and  prioritize  diagnostic  measures  and  metrics  that  can  determine 
whether  a  program  or  acquisition  job  is  “likely  staying  on  track” 

Set  up  near-term  implementation  described  on  previous  slides 

-  Start  by  distributing  models  and  diagnostic  metrics,  identifying  key 
personnel  who  should  be  involved  in  the  “ASAP  for  Acquisition” 
network,  and  identifying  key  sources  of  acquisition  data 

Begin  building  a  library  of  decision  rules  on  how  to  flag  information  for 
further  review,  and  what  sorts  of  hypotheses  to  generate 


Unclassified 


For  More  Information... 


•&  Out  of  the  Ordinary:  Finding  Hidden 
Threats  by  Analyzing  Unusual  Behavior 
available  at: 

-  http://www.rand.org/publications/MG/ 
MG126/ 


•  POC:  John_Hollywood@rand.org 


RAND 
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Mission  Sustainment  Through 
Environment,  Safety,  and  Occupational 
Health  (ESOH)  Risk  Management 


Systems  Engineering  Conference 
San  Diego,  CA 
October  24  -  27,  2005 


Trish  Huheey 

Office  of  the  Deputy  Under  Secretary  of  Defense 
(Installations  &  Environment) 


Table  of  Contents 


•  Increased  DoD  Emphasis  on  ESOH 

•  Why  Integrate  ESOH  in  Systems 
Acquisition 

•  Key  Initiatives  to  Enhance  ESOH 
Integration 

•  ESOH  Risk  Management  with 
System  Safety 

•  ESOH  Risk  Management  is  not 
Optional 

•  Other  DoD  Sustainment  Related 
Policies  With  Acquisition  ESOH 
Considerations 

•  Conclusions 
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Increased  DoD  Emphasis  on  ESOH 


•  Total  Life-Cycle  Systems  Management  (TLCSM) 

-  PM  is  the  designated  life-cycle  manager 

-  Conduct  trade  analyses  early  to  achieve  savings  and  ensure 
ESOH  compliance  throughout  the  life-cycle,  especially  regarding 
impacts  to  mission  sustainment  and  installations 

•  Systems  Engineering  (SE)  and  Performance  Based 
Logistics  (PBL)  strategies 

-  Increased  reliability  and  reduced  logistics  footprint 

-  ESOH  risk  reduction  and  effective  integration  into  systems 
engineering  process 

•  AT&L  Memorandum  on  Policy  for  Systems  Engineering  in 
DoD,  20  February  2004  -  Systems  Engineering  Plan 

-  Mandates  robust  SE  approach  that  balances  total  system 
performance  and  total  ownership  costs 
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Why  Integrate  ESOH  in 
Systems  Acquisition? 

•  ESOH  regulations  potentially  impact  sustainability  of 
operations,  maintenance,  and  natural  infrastructure 

•  ESOH  compliance  issues  can  impact  capability  to  achieve 
system  missions  and  exacerbate  encroachment  problems 

•  DoDD  and  DoDI  5000.2  policies 

-  Program  Mangers  required  to  identify,  eliminate  when  possible,  and 
minimize  ESOH  hazards  and  associated  risks 

-  ESOH  considerations  are  integral  part  of  defense  system’s 
operational  effectiveness  and  sustainability 

-  SE  is  one  of  the  primary  processes  for  addressing  ESOH  with 
regard  to  system  design,  test,  deployment,  operation,  training, 
maintenance,  and  disposal,  as  well  as  to  help  maximize  resources 
and  reduce  total  life-cycle  costs. 


ESOH  Risk  Management  Is  Not  Optional 


•  ESOH  risk  management  includes: 

-  Hazardous  materials  and  waste 

-  Environmental  and  occupational  noise 

-  Personnel  safety  and  occupational  health 

-  Natural  environmental  assets  and  infrastructure 

-  Compliance  with  numerous  regulations 

-  System  safety  and  explosive  safety 

•  ESOH  risk  management  must  be  ingrained  in  the 
program’s  entire  risk  identification  and  acceptance 
process 

-  Using  a  risk  management  approach  for  ESOH  compliance  does 
not  allow  programs  to  accept  risks  of  non-compliance  with  ESOH 
regulations;  it  does  highlight  the  impacts  and  allow  for  informed 
decision-making 
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ESOH  Risk  Management  Is 
Not  Optional  (Cont.) 


•  Lack  of  attention  or  failure  to  address  ESOH 

requirements  and  risks  early  in  the  system  acquisition 
process  could  impact  program  cost,  schedule, 
performance,  and  result  in 

-  Loss  of  life  and/or  serious  injury  to  personnel 

-  Serious  damage  to  facilities  and/or  equipment 

-  Serious  adverse  impact  on  mission  capabilities  and  operability 
from  failures 

-  Negative  public  opinion 

-  Potential  lawsuits  or  injunctions  from  citizens  and  public  interest 
groups 

-  Detrimental  harm  to  the  environment  and  surrounding 
community 


Key  Initiatives  to  Enhance  ESOH  Integration 


•  Revise  DAU  curricula  to  improve  ESOH  content 

-  DAU  System  Safety  in  Systems  Engineering  Continuous  Learning 
Module  (April  2005) 

-  Acquisition  Core,  System  Engineering,  Logistics,  Test  and 
Evaluation,  and  Program  Management  functional  areas 

•  Establishment  of  ESOH  risk  management  criteria  in  the 
acquisition  process 

•  Incorporation  of  ESOH  considerations  in  the  Systems 
Engineering  Plan  (SEP) 
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ESOH  Risk  Management 
With  System  Safety 


•  System  safety  is  the  SE  approach  for  eliminating  or 
minimizing  ESOH  hazards  and  risks  across  the  system’s 
life-  cycle 

•  AT&L  Memorandum  on  Defense  Acquisition  System 
Safety,  23  September  2004 

-  Integration  of  system  safety  into  SE  mandated 

-  MIL-STD-882D,  DoD  Standard  Practice  for  System  Safety ,  used 
for  ESOH  risk  identification  and  management 

•  Defense  Safety  Oversight  Council  (DSOC),  July  2003 

-  Acquisition  and  Technology  Programs  Task  Force,  April  2004 

-  Investigate  and  recommend  investments  to  ensure  safety  is 
addressed  throughout  an  acquisition  program’s  life-cycle 


8 


System  Safety  Order  of  Precedence 


Listed  in  order  from  most  to  least  preferred 


1.  Eliminate  hazards  through  1. 

design  selection 

2.  Incorporate  safety  devices  2. 

3.  Provide  warning  devices  3. 

4.  Develop  procedures  and  4 

training 


If  unable  to  eliminate  an  identified 
hazard,  reduce  the  associated  risk  to  an 
acceptable  level  through  design 
selection. 

If  unable  to  eliminate  an  identified 
hazard  through  design  selection,  reduce 
the  risk  to  an  acceptable  level  using 
protective  safety  features  or  devices. 

If  safety  devices  do  not  adequately 
lower  the  risk  of  the  hazard,  include  a 
detection  and  warning  system  to  alert 
personnel  to  a  particular  hazard. 

Where  it  is  impractical  to  eliminate 
hazards  through  design  selection  or  to 
reduce  the  associated  risk  to  an 
acceptable  level  with  safety  and 
warning  devices,  incorporate  special 
procedures  and  training.  Procedures 
may  include  to  use  of  protective 
equipment. 


Risk  Mitigation  Measures  (RMM)  Example 


PROBABILITY 

LEVEL 


(A)  Frequent 


(B)  Probable 


(C)  Occasional 


(D)  Remote 


(E)  Improbable 


Prc 


SEVERITY  CATEGORY 

i 

CATASTROPIC 

ii 

CRITICAL 

in 

MARGINAL 

IV 

NEGLIGIBLE 

1 

3 

7 

13 

2 

5 

9 

16 

cedures^j  / 

11 

18 

t/  8 

Warnings 

10  ^ 

19 

12  ^ 

Saf( 

r 

L 

ity 

15 

Devices 

Design  ^ 

Change  y 

^\20 

High  Risk 
Serious  Risk 


Medium  Risk  A  Initial  Risk 
Low  Risk  A  Residual  Risk 


Hazard  Eliminated 


Conclusions 


•  Effective  ESOH  integration  into  the  acquisition  SE 
process  is  essential  to  sustainability  and  operational 
effectiveness 

-  Regulatory  and  public  interest  concerns  can  impede  ability  to 
develop,  test,  operate,  maintain,  train,  and  dispose  of  systems 

-  Effective  life-cycle  ESOH  risk  management  during  system  design 
and  development  maximizes  potential  for  cost  and  liability 
reduction 

•  The  PM  must  look  forward  over  the  entire  life-cycle  of  the  system  and 
make  the  case  to  Resource  Sponsors  for  early  investments  for  future 
cost  avoidance 

•  Resource  sponsors  must  recognize  that  investing  in  designs  and 
materials  that  reduce  the  burden  on  the  natural  infrastructure  result  in 
lower  life-cycle  costs  and  unencumbered,  sustainable  operations 

-  Further  supports  the  mandatory  use  of  TLCSM 
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Questions  and  Answers 


Ms.  Trish  Huheey 
ODUSD(l&E) 

3400  Defense  Pentagon 
Washington,  DC  20301-3400 

703-604-1 846  DSN225 
Patricia.huheey@osd.mil 


Acquisition  ESOH  Guidance  References 

Acquisition  Community  Connection,  ESOH 
Special  Interest  Area  (http://acc.dau.mil) 

AT&L  Knowledge  Sharing  System  (AKSS) 
( http  ://deskbook.dau  .mil) 
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